RL @ ECE-UofT - Lecture 00: Course Overview and Logistics

Overview

Ali Bereyhi outlines the reinforcement learning course at ECE-UofT, progressing from environment models and dynamic programming to model-free methods such as SARSA and Q-learning, then deep RL with value networks, policy gradients, and actor-critic approaches. He also explains course logistics, four assignments, two exams, a group project, programming tools, and rules for responsible AI use.

Key takeaways

The course moves from model-based RL, where environment dynamics are known and Bellman methods can find optimal actions, to model-free learning from interaction data.
The deep RL portion uses neural networks to address the computational limits of tabular methods and covers value-based learning, policy gradients, and actor-critic systems.
Students complete four assignments, two one-hour exams on October 23 and December 4, and a group-of-three project instead of a final exam.
AI can assist with learning, debugging, and project research, but students must not submit AI-generated assignment solutions or project reports and must attest to their AI use.
Python, PyTorch, NumPy, and Gymnasium are the main programming tools; prior experience training neural networks is expected for the course's later deep RL material.

Chapters

0:00 Course Logistics: GitHub, Quercus, Crowdmark, and Piazza

Ali Bereyhi says the course meets once weekly from 1 to 4 p.m.; Friday tutorials begin the following week and focus on implementation.
Course materials are hosted on GitHub, while Quercus sends announcements and Crowdmark handles assignment submissions.
Students should post each new question as a separate Piazza thread; email Ali with the prefix EC1508 RL when needed.

5:30 The Course Progression from RL Foundations to Deep Learning

The opening portion introduces the reinforcement learning problem, environments, and mathematical formulations, before moving to model-based and model-free methods.
Model-based methods assume knowledge of environment statistics; practical model-free learning instead uses data gathered from the environment.
Deep RL addresses computational scale by using neural networks for function approximation, with value-based, policy-gradient, and actor-critic algorithm families.
Students need basic machine learning experience and should be able to implement and train neural networks; prior deep learning course equivalents are suitable preparation.

14:15 Algorithms on the Course Map: Bellman Methods, SARSA, and Q-Learning

The model-based section covers Bellman equations and optimality, value iteration, and policy iteration when environment dynamics are known.
Model-free topics include on-policy learning with temporal-difference and Monte Carlo methods, including SARSA.
Off-policy learning leads to Q-learning, while the later deep RL section covers value networks, policy gradients, and actor-critic approaches.

17:30 Four Assignments and the Rules for AI-Assisted Work

The course has four assignments, released roughly every two weeks, combining conceptual and programming questions; unsubmitted practice questions support exam preparation.
The first assignment covers RL formulation, the second covers tabular methods, and the final two focus on deep reinforcement learning.
AI tools may clarify concepts, interpret prompts, debug code, or support sanity checks, but AI-generated solutions must not be submitted.
Each assignment requires an AI-use attestation describing how AI was used.

21:00 Two Exams and a Three-Person Reinforcement Learning Project

Two one-hour exams are scheduled for October 23 and December 4, each with two questions based on course, assignment, and practice material.
The final project replaces a final exam; groups of three may choose a research or design project and submit a proposal by October 12.
Project proposals include group members and a Git repository link; private repositories must add Ali or a course team member as a collaborator.
Groups give a five-minute presentation during December 7 to 11 and submit the report and code by December 18.

27:48 Python, Gymnasium, Sutton and Barto, and Asking Questions

Assignments use Python, with PyTorch and NumPy for neural networks and Gymnasium for standardized and predefined RL environments.
Lecture notes provide the main preparation materials; the course draws on Richard Sutton and Andrew Barto's reinforcement learning textbook.
Ali may revise slides before a lecture and may release notes in batches, with the relevant materials available through the course GitHub page.
Students are encouraged to ask questions, including basic ones, especially when RL notation, time indices, and formulation details become difficult.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Ali Bereyhi.