Stanford AA203 Optimal and Learning-Based Control | Spring 2026 | Lecture 15: Imitation Learning

Overview

This lecture from Stanford Online's AA203 course delves into imitation learning, focusing on behavior cloning and its challenges like compounding errors and multimodal behavior. Strategies to address these include algorithms like DAgger for corrective data collection, data augmentation techniques (e.g., NVIDIA's autonomous driving), and using expressive models like Gaussian Mixture Models or autoregressive approaches to capture complex distributions. The lecture also touches upon action chunking and diffusion models for generating trajectories.

Key takeaways

Behavior cloning's naive approach suffers from compounding errors (covariate shift) and multimodal behavior, requiring advanced techniques.
DAgger iteratively collects corrective data by querying experts on states visited by the learner policy, directly addressing state distribution mismatch.
Data augmentation, like NVIDIA's use of side cameras for autonomous driving, can fictitiously generate corrective driving data without real-world risk.
Expressive models like Gaussian Mixture Models, autoregressive transformers, and diffusion models are crucial for capturing complex, multimodal action distributions in robotics.
Action chunking, predicting a sequence of future actions, leads to smoother control trajectories and is a popular parameterization for robot learning policies.
Inverse Reinforcement Learning (IRL) aims to recover the expert's underlying reward function, addressing the ambiguity where multiple rewards can explain observed behavior.

Chapters

0:00 Introduction to Imitation Learning and Roadmap

Transition from optimal control to learning-based methods.
Overview of behavior cloning (BC) and inverse reinforcement learning (IRL).
Focus on BC: learning a policy from expert demonstrations (state-control pairs).

1:56 Behavior Cloning as Supervised Learning

BC framed as supervised learning: mapping inputs (states) to outputs (controls).
Objective: learn a parameterized policy pi(theta) from expert data.
Data set consists of (state, control) pairs from expert demonstrations.

5:01 Pitfall 1: Compounding Errors and Covariate Shift

Problem: state distribution induced by learner policy diverges from expert's.
Small errors in early steps can compound over time, leading to larger deviations.
Formalized as covariate shift; error probability grows quadratically with trajectory length.

16:41 Pitfall 2: Multimodal Behavior

Occurs when multiple equivalent solutions exist for a task (e.g., drone passing tree left or right).
Standard loss functions (like MSE) can lead to learning the mean of the distribution, failing to capture distinct modes.
Problem exacerbated when data comes from multiple experts with different behaviors.

20:52 Strategies for Addressing Pitfalls: Algorithms

Focus on algorithms, data collection, and expressive model classes.
Value of corrective data: learning from expert mistakes and recovery is crucial.
DAgger (Dataset Aggregation) algorithm addresses covariate shift iteratively.

27:10 DAgger Algorithm Explained

Iterative process: collect trajectories with current policy, query expert for relabeling in visited states.
Aggregate new expert labels with existing data to train the next policy iteration.
Queries expert on states visited by the learner, directly addressing state distribution mismatch.

34:17 Human-Gated DAgger and Extensions

Human-gated DAgger allows human intervention to generate new trajectory continuations after mistakes.
Iterative nature of DAgger covers new state distributions over iterations.
DAgger is data-efficient for querying expert on relevant states.

43:56 Addressing Covariate Shift via Data Collection: NVIDIA Driving

Case study: NVIDIA's early work on autonomous driving using camera images and steering angles.
Challenge: collecting dangerous driving mistake data.
Solution: data augmentation using side cameras to fictitiously generate corrective steering angles.

50:35 Data Collection: University of Zurich Quadrotor Navigation

Case study: controlling quadrotors using camera images to navigate hiking trails.
Method: researchers wore head-mounted cameras (GoPros) to collect data.
Augmentation: labeling front camera as 'go straight', side cameras as 'turn left/right'.

53:44 Intentional Data Collection for Corrective Behavior

Main message: be intentional with data collection to address covariate shift.
Add mistakes and corrections to the data set, either via augmentation or algorithms like DAgger.
Imitation learning approximates expert behavior; cannot discover behaviors beyond expert's capabilities.

58:29 Expressive Models for Multimodal Behavior: History

Problem: partial observations (e.g., single image) may not fully represent state.
Solution: include a history of past observations in the policy's input.
Handles situations where same observation can lead to different actions based on prior context.

1:05:03 Expressive Models for Multimodal Behavior: Distribution Representation

Goal: represent the full, potentially multimodal, data distribution, not just the mean.
Discrete distributions (e.g., categorical) naturally handle multimodality but scale poorly with dimensionality.
Continuous distributions (e.g., Gaussian) are efficient but struggle with multimodality (e.g., fitting bimodal data with unimodal Gaussian).

1:13:21 Gaussian Mixture Models (GMMs) and Discretized Autoregressive Models

GMMs: mixture of k Gaussians to approximate complex, multimodal distributions.
Discretized Autoregressive: decomposes joint distribution into conditional 1D distributions, reducing dimensionality issues.
Transformer models often use this autoregressive approach for robot policies.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Stanford Online.