COMP 3200 - Intro to Artificial Intelligence - Lecture 15 - Intro to Reinforcement Learning
Overview
Dave Churchill introduces reinforcement learning as learning through interaction: an agent observes states, takes actions, receives rewards, and updates its policy to maximize expected future return. He contrasts reinforcement learning with supervised and unsupervised learning, explains tabular policies, reward and value functions, and uses CartPole, Mountain Car, pathfinding, and blackjack to make the concepts concrete. The lecture closes with exploration versus exploitation and notes that deep reinforcement learning replaces a tabular representation with a neural network.
Key takeaways
- Reinforcement learning optimizes expected cumulative reward, not just immediate reward, so a temporarily pleasant action can be undesirable if it leads to worse future outcomes.
- Reward design is part of problem formulation: +1 per balanced CartPole time step rewards endurance, while −1 per Mountain Car time step encourages a faster goal-reaching policy.
- A value function estimates long-term return from a state or action, while the reward function supplies the immediate environmental feedback used to learn.
- Reinforcement learning can operate without a simulator or explicit physics model, unlike search methods such as A* and minimax, but it needs a training phase and may require retraining when the environment changes.
- Exploration is necessary to discover better actions, while exploitation uses known high-value actions; epsilon-greedy methods make this balance explicit by choosing random actions with probability ε.
Chapters
- Dave Churchill recommends Sutton and Barto’s free textbook, Reinforcement Learning: An Introduction, for deeper study.
- The course focuses on tabular reinforcement learning, where a policy maps states to actions using a table that fits in memory.
- Deep reinforcement learning scales beyond a table by using a neural network to approximate policies across many more states.
- Supervised learning trains on input-output examples, such as fruit images paired with labels like tomato or banana.
- Classification assigns inputs to discrete classes, as in a cat-versus-dog image model or a support vector machine separating data regions.
- Regression approximates numerical outputs, as in fitting a line to data or learning the pattern 1→1, 2→4, 3→9.
- Unsupervised learning finds structure in unlabeled data; clustering groups similar handwritten digit images without supplied labels.
- In k-means, choosing k = 10 can produce ten clusters corresponding roughly to the digits zero through nine.
- Visualizing clusters reveals overlaps, outliers, and ambiguous handwriting, such as a seven written in a way that resembles a four.
- Reinforcement learning is a problem framework, not a single algorithm: a problem is represented using states, actions, an environment, and rewards.
- An agent learns by acting and receiving environmental feedback rather than by being given the correct action for every state.
- Touching a hot stove illustrates negative reinforcement: the painful outcome makes repeating that action less desirable in a similar state.
- At time step t, the agent observes state sₜ, selects a legal action aₜ, receives a real-valued reward, and transitions to state sₜ₊₁.
- A policy determines which action to take from a state, while generalized policy iteration describes improving the policy as the agent learns from experience.
- For games such as Super Mario Bros., the programmer must translate the objective into a reward signal, for example rewarding movement to the right.
- For CartPole, awarding +1 for every balanced time step makes the episode return reflect how long the pole stays upright.
- Mountain Car requires rocking left and right to build momentum; a −1 reward per time step encourages reaching the goal in fewer steps.
- These examples show how reward design turns an English objective, such as balance or quick escape, into a numerical learning signal.
- A reinforcement learning problem includes an agent, environment, reward function, policy, and value function; an environment model is optional.
- Unlike search methods such as A* and minimax, model-free reinforcement learning can learn from states and rewards without simulating the environment’s physics.
- Search can run immediately and adapt to changed rules, but needs an environment model; learning can avoid that model but requires training and may need retraining after changes.
- A policy maps states to actions and may be deterministic or stochastic, such as choosing action A 20% of the time and action B 80% of the time.
- In a grid environment, a learned policy can specify directions toward the nearest goal and assign equal probabilities to multiple equally good moves.
- A blackjack policy table maps a player’s hand and dealer’s visible card to actions such as hit or stand; optimal play still loses more hands than it wins in the example given.
- The reward function converts the task goal into a number for a state or state-action pair, so the same action can be good in one state and dangerous in another.
- The return is the sum of future rewards, and the agent aims to maximize expected return rather than a single immediate reward.
- Blackjack outcomes are stochastic, so the optimal policy maximizes average results over repeated hands, not a guaranteed win on any particular hand.
- Card counting changes betting based on the remaining deck and dealer’s fixed rules; it does not change the optimal hit-or-stand policy.
- For an episodic task, return Gₜ sums rewards from time t through the episode’s final step T.
- An optimal policy π* maximizes expected return, which means selecting the best long-run average outcome available in the environment.
- Even an optimal blackjack policy cannot guarantee profit; optimality means doing the best possible under the game’s probabilities.
- A value function estimates the expected future return from a state or state-action pair, unlike a reward, which is observed immediately.
- A pathfinding state on a short route to the goal should have higher value than one on a longer route, because it predicts better future returns.
- An environment model predicts the next state from a state and action; simulation can support training, but reinforcement learning does not always require a model.
- Exploitation repeats actions already known to work, while exploration tests alternatives that may reveal better rewards.
- Choosing where to eat illustrates the tradeoff: returning to a familiar restaurant is reliable, but trying a new one can uncover a better option at a cost.
- An epsilon-greedy method takes a random action with probability ε, such as exploring 10% of the time while otherwise choosing the currently preferred action.
- Game-playing agents can learn from human replays before self-play, using winning and losing games to provide useful experience before exploring from scratch.
Transcript excerpt
0:00 All right. So lecture number 15, we're going to start our adventure into reinforcement learning, which is one of my favorite parts of this class, and I think is one of the more interesting things especially because recently reinforcement learning has just been it's become incredibly popular and incredibly powerful, right? Perhaps more so than search, at least in in specific domains. And so I will give a shout out to the optional. I say it's an optional textbook, but it's also a free textbook, and it's available online. So the textbook is called Reinforcement Learning An Introduction. I can't remember what the most current version is, but it was written by Sutton and Barto Barto, apologies, and it's the best textbook I've ever read. And so if you are
0:49 studying for the final or you at any time want to get more information or you're interested about something, please read the textbook. What I did to make these slides is go along with the textbook and then pick out the parts that I thought were more interesting. Just another quick note is that there is a mathematics course on reinforcement learning. Has anyone taken or enrolled in that class? So there's there's a class in the math department on reinforcement learning, and someone in a previous class who had already done a math degree told me that we basically do that whole course in like six lectures. But the math degree goes way heavier on the math, which I don't really like anyway. So we're going to do the practical implementation. Here's just how it works and how we can implement it sort of
1:36 thing. We'll leave all the all the mathematical details for the math course and for the textbook. And so please please have a look at the textbook. And
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Dave Churchill.