RL @ ECE-UofT - Lecture 09: Policy Gradient Theorem and PGMs
Overview
Ali Bereyhi derives the policy gradient theorem for parameterized stochastic policies, showing how the gradient of expected return can be estimated from sampled state-action pairs using action values and the gradient of the policy’s log likelihood. He develops vanilla policy gradient and REINFORCE, explains why reward shifts can destabilize them, then derives action-independent baselines and the advantage function, connecting advantage estimates to TD(0) and n-step returns.
Key takeaways
- The policy gradient theorem makes policy optimization possible without differentiating through the environment: estimate an action value and multiply it by ∇θ log πθ(a|s).
- For a sampled trajectory, the gradient of its log probability reduces to a sum of policy-network log gradients because the MDP transition probabilities do not depend on θ.
- Adding a constant to rewards can inflate vanilla policy-gradient updates even when the optimal action ranking is unchanged, since returns directly scale the update.
- Any baseline U(s) that depends on state but not action preserves the expected policy gradient because Eπ[∇θ log πθ(a|s)] = 0.
- Using Vπ(s) as the baseline yields the advantage Aπ(s,a) = Qπ(s,a) − Vπ(s), which measures how much an action improves on the policy’s average outcome at that state.
- A TD(0) advantage estimate, rₜ₊₁ + γV(sₜ₊₁) − V(sₜ), needs only a value estimator; actor-critic supplies one alongside the policy and avoids requiring an explicit Q network.
Chapters
- A value network can be trained by matching its prediction to Monte Carlo or TD value estimates with squared error.
- An optimal policy is defined by maximizing value, but directly turning that definition into a policy-network loss is not obvious.
- The policy gradient theorem supplies a usable gradient based on the log likelihood of sampled actions.
- For discrete actions, the policy network outputs one probability per action, with a softmax ensuring nonnegative values that sum to one.
- For continuous actions, a network can output a Gaussian mean and variance, then sample an action from that distribution.
- A deterministic policy is possible, but stochastic policies are more general and can approach deterministic behavior as variance shrinks.
- A particle’s position, such as coordinates x and y, serves as its state representation.
- With three discrete moves, the network outputs a three-entry softmax for left, up, and right.
- With continuous movement direction, the network can produce distribution parameters and sample an angle.
- A linear discrete policy multiplies state features by a parameter matrix, then applies softmax to produce action probabilities.
- The parameter matrix has one row per action and one column per state feature; an MLP or CNN can replace the linear model.
- A continuous linear policy can map state features to a Gaussian mean, with variance either fixed or learned.
- The goal is to choose parameters θ that maximize policy value, rather than optimize value at just one state.
- Because one shared θ affects every state, optimizing one state can harm performance elsewhere, creating a multiobjective problem.
- A common scalarization averages state values under a distribution over states.
- The probability of visiting a state depends on the environment’s transitions, the policy, and how initial states are sampled.
- Changing θ changes action probabilities and therefore changes the distribution of later states.
- Ali Bereyhi uses a sampled initial-state distribution to define an expected-return objective over trajectories.
- For an episodic task, a trajectory contains states, actions, rewards, and a terminal state.
- With discount factor γ set to 1 for the initial derivation, the return G₀ is the sum of rewards along the episode.
- The value at a fixed starting state is the expected return over all trajectories generated by the policy and MDP.
- A trajectory probability factors into policy action probabilities and MDP transition probabilities.
- Fixing S₀ does not fix later states or actions, since both the policy and environment can be stochastic.
- The expected return is obtained by weighting each possible trajectory’s return by its probability.
- Averaging value over initial states incorporates the chosen distribution for where episodes begin.
- Once the initial state is sampled, the policy and environment determine the rest of the trajectory distribution.
- The objective can be written as an expectation of trajectory return under a distribution that depends on θ.
- Differentiating expected return moves the gradient inside the trajectory expectation because integration and differentiation can be exchanged under the setup.
- The return for a fixed trajectory does not directly depend on θ; the trajectory probability does.
- Using ∇p = p∇log p converts the gradient into an expectation of return times the trajectory log-likelihood gradient.
- The trajectory log probability is a sum of log policy probabilities and log MDP transition probabilities.
- The transition model and initial-state distribution do not depend on policy parameters θ, so their gradients vanish.
- The remaining gradient is a sum of ∇θ log πθ(aₜ|sₜ), computable by backpropagation through the policy network.
- A sampled episode provides a return and a sum of log-policy gradients, yielding one sample gradient estimate.
- A batch of K trajectories can estimate the gradient by averaging their return-weighted gradients.
- Stochastic gradient ascent updates θ using the estimated gradient and a learning rate.
- Using one trajectory at a time gives a high-variance but simple stochastic policy-gradient update.
- REINFORCE applies a return from time t onward, Gₜ, to the log-policy gradient at state-action pair (sₜ, aₜ).
- Treating each time step as a sample explains why REINFORCE uses Gₜ rather than multiplying every action gradient by the episode’s G₀.
- The theorem expresses the objective gradient as an expectation of Qπ(s,a) times ∇θ log πθ(a|s), under policy-induced state-action visitation.
- A sampled return from time t onward, Gₜ, is a Monte Carlo estimator of Qπ(sₜ,aₜ).
- The state-action expectation can be approximated by sampling points along trajectories and averaging their estimates.
- REINFORCE is a vanilla policy-gradient method because it uses sampled returns to estimate action values.
- Actor-critic algorithms use the same log-policy-gradient structure but estimate value terms with a critic.
- Ali Bereyhi identifies policy-gradient methods as a broad family, contrasting them with value-based approaches such as Q-learning and SARSA.
- A vanilla policy-gradient update multiplies an estimate of Q by the gradient of the log policy.
- Monte Carlo estimation computes Q at time t by summing the subsequent rewards, optionally with discounting.
- Using only the policy network leaves no learned value estimator for TD bootstrapping.
- In the particle example, moving straight upward is optimal, and rewards are highest near the center of the state space.
- Adding a constant shift of 11 preserves which actions are better, but changes rewards from roughly -1 to 1 into roughly 10 to 12.
- With shifted rewards, the vanilla policy-gradient run learns much more slowly because the return-weighted updates become larger.
- The Q estimate scales the policy update, so adding a reward offset can act like increasing the effective learning rate.
- Large parameter updates can sharply alter action probabilities, even when the underlying reward ranking has not changed.
- Subtracting a state-dependent reference from Q can recenter the update signal without changing the desired gradient.
- Define Q′(s,a) = Qπ(s,a) − U(s), where the baseline U depends on state but not on action.
- The extra gradient term contains U(s) times the policy expectation of ∇θ log πθ(a|s).
- That policy expectation is zero, so replacing Q with Q − U leaves the expected policy gradient unchanged.
- For any normalized parameterized distribution Pθ, the expectation of ∇θ log Pθ(x) equals the integral or sum of ∇θPθ(x).
- Because the probabilities sum or integrate to 1, their gradient sums to ∇θ1 = 0.
- This identity makes any action-independent state baseline valid for policy-gradient estimation.
- Choosing U(s) = Vπ(s) compares each action’s outcome with the expected value of being in that state.
- The resulting advantage is Aπ(s,a) = Qπ(s,a) − Vπ(s).
- Positive advantage indicates an action performs better than the policy’s state-level average; negative advantage indicates worse performance.
- An advantage-based update multiplies Aπ(s,a) by ∇θ log πθ(a|s).
- Subtracting Vπ(s) removes a shared state-level offset, helping reduce the impact of large positive returns.
- The value baseline is a useful standard choice, though its role is to preserve the expected gradient while improving the numerical signal.
- A TD estimate of Qπ(sₜ,aₜ) uses the immediate reward plus γVπ(sₜ₊₁).
- A pure policy-gradient method has no value table or value network to supply Vπ(sₜ₊₁).
- Without a critic, REINFORCE therefore relies on Monte Carlo returns, which can be high variance and require completed episodes.
- Monte Carlo can estimate returns directly, but may be inefficient, especially in long or non-episodic environments.
- Repeatedly evaluating each updated policy from scratch is impractical because every policy change requires new value estimates.
- Actor-critic adds a value network trained alongside the policy, allowing the critic to estimate values for policy updates.
- The actor is the policy network that selects actions, while the critic estimates state values.
- Policy-generated samples train the critic in parallel with policy updates.
- With Vπ available, the advantage can be estimated without building a separate action-value network, an important simplification for continuous actions.
- Bellman’s equation writes Qπ(sₜ,aₜ) as the expected immediate reward plus γVπ(sₜ₊₁).
- Replacing expectations with one sampled transition gives the advantage estimate rₜ₊₁ + γV(sₜ₊₁) − V(sₜ).
- This quantity is the TD(0) error, now used as a sample estimate of advantage for policy updates.
- The TD(0) advantage estimate needs values for the current and next states, not an explicit Q network.
- Estimating V(s) is especially convenient when actions are continuous, because evaluating Q over all actions can be difficult.
- The idealized value function in the derivation can be replaced in practice by a learned value network.
- An n-step estimate sums rewards for n transitions, then adds a bootstrapped value estimate at the state reached after n steps.
- Subtracting V(sₜ) from that n-step target yields another estimator of Aπ(sₜ,aₜ).
- The same idea can be extended toward TD(λ), combining information from different rollout lengths.
- Using an advantage instead of an uncentered return can make policy updates more stable and less sensitive to reward shifts.
- For simpler environments, sample returns can estimate the values; more complex settings benefit from a learned value network.
- The lecture notes that advantage methods can converge faster in the particle-navigation example.
- A single sampled policy trajectory contains many state-action samples that can contribute to gradient estimation.
- The lecture begins examining how effectively those samples can be reused, identifying sample efficiency as a remaining weakness.
- The closing point sets up further discussion of how policy-gradient algorithms use trajectory data.
- One trajectory generated by πθ may contain many observations, rather than only one usable training sample.
- Those state-action pairs can provide estimates of the policy gradient through action-value or advantage targets.
- The lecture ends by framing sample reuse as a key concern for policy-gradient methods.
Transcript excerpt
0:00 It was quite simple to train it, right? So you had the output of your neural network which was value and then you wanted to make this exactly like the value of this state and then you just instead of the value you sample either using Monte Carlo or TD you just sample that one you take the estimator of the value and you try to match it that was quite simple you just use listed squares and you train it right but in this case it's quite unclear what should be the training approach so I have a policy network I want it to behave like the optimal policy right but what is the optimal policy okay that is the guy which is uh having the maximum value right but how you can interpret this one into training loop that is a bit
0:49 not say straightforward and uh the whole trick of the policy gradient method theorem is that you can show that actually as long as you compute a gradient which is proport proportional to logarithm of the policy network. So log likelihood of the of the policy or the actions that you do then that is a valid gradient to train your neural network. Once you understand that one then the rest of it is again developing tricks and then we're going to see different challenges when we try to come over with with different tricks. Okay. So let's start with this idea of policy network. So the policy network as I told you last time is a neuronal network which is going to take as input your
1:37 state and of course you do know that you're not giving this state itself you're going to give the representation of the state right so in the example of
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ali Bereyhi.