Princeton Intro to Robotics (Fall 2026) | Lecture 9: Optical Flow

Overview

Optical flow estimates the apparent motion of image features and can help a drone infer its velocity, complementing noisy position estimates from integrated accelerometer readings. The lecture derives the Lucas-Kanade method from intensity constancy and small-motion assumptions, explains the aperture problem and spatial-coherence assumption, and shows how least squares estimates flow while failing on textureless regions or single edges.

Key takeaways

A downward-facing drone camera can estimate motion relative to the ground from optical flow; combined with a height measurement, that flow can provide speed and direction.
Optical flow measures apparent image motion: a textureless surface can hide real movement, while changing illumination can create apparent movement without physical motion.
The optical-flow constraint Ix u + Iy v + It = 0 supplies only one equation for two motion components, leaving motion perpendicular to the image gradient unobservable.
Lucas-Kanade addresses that ambiguity by assuming a local window, often 5×5 or 7×7 pixels, shares one flow vector and solving its overdetermined constraints with least squares.
Lucas-Kanade fails or becomes unstable in textureless regions and along single edges because the local image gradients do not provide independent directional information.
Bird corridor experiments and honeybee navigation illustrate biological uses of optical flow: birds adjust speed and position based on perceived motion, while bees encode route distance through accumulated flow.

Chapters

0:00 Why Drone Localization Needs More Than Accelerometers

The course shifts from motion planning and feedback control to sensing, state estimation, localization, and mapping.
A drone can integrate accelerometer measurements twice to estimate position, but sensor noise compounds through integration.
A downward-facing optical-flow camera measures apparent scene motion and can help the drone correct deviations from planned waypoints.

4:00 Reading Drone Motion from an Optical-Flow Field

An optical-flow field assigns image locations vectors indicating the apparent direction and magnitude of motion.
In the forward-camera example, the flow pattern suggests the drone is primarily turning left.
With a downward-facing pinhole camera and a height measurement, optical flow can estimate the drone's speed and direction relative to the surface.

8:46 Apparent Motion Is Not Always Physical Motion

Optical flow describes perceived image motion, not necessarily the true motion of the camera or objects.
A drone moving over a perfectly textureless carpet may experience physical motion without measurable optical flow.
Changing illumination can create apparent optical flow even when the camera and environment are physically still.

12:30 How Birds Use Optical Flow to Navigate Corridors

Corridor experiments found that birds flew more slowly beside vertical stripes, which generated stronger apparent motion.
With vertical stripes on one wall and horizontal stripes on the other, birds flew closer to the horizontal-striped wall, apparently balancing flow between their eyes.
When the opposite wall was blank, birds sometimes collided with it, showing that flow-based navigation depends on visual texture.

16:12 Honeybee Waggle-Dance Distance Uses Integrated Flow

A honeybee's waggle duration indicates distance to nectar, while its dance angle communicates direction relative to the sun.
The distance signal reflects accumulated optical flow along the route, rather than a direct measurement in meters.
Changing visual texture along a bee's route altered how far another bee traveled while retracing it.

20:52 Optical-Flow Computation Starts with Two Assumptions

The model-based method estimates a motion vector for each image pixel from consecutive grayscale frames.
Intensity constancy assumes a moving image patch retains approximately the same appearance between frames.
The small-motion assumption requires image features to move only a little between frames, enabling a first-order approximation.

24:57 Deriving the Optical-Flow Constraint Equation

For pixel location (x, y), intensity constancy is written as I(x, y, t) = I(x + u, y + v, t + 1), where (u, v) is flow.
A first-order Taylor expansion produces the constraint Ix u + Iy v + It = 0.
Spatial and temporal derivatives can be estimated from neighboring pixels and consecutive frames using finite differences.

31:37 The Aperture Problem: One Equation Cannot Recover Two Components

The optical-flow constraint gives one scalar equation for the two unknowns u and v at each pixel.
Motion perpendicular to the local image gradient is unobservable from that equation alone.
The barber-pole-style aperture demonstration shows how perceived motion changes when the viewed region expands, revealing the need for additional assumptions.

37:43 Why Local Image Features Leave Motion Ambiguous

Adding any motion component perpendicular to the image gradient preserves the optical-flow constraint, so that component cannot be uniquely recovered.
The ambiguity is fundamental to the local intensity-based equation, not merely a weakness of a particular numerical solver.
Resolving it requires assumptions about the scene or neighboring pixels beyond intensity constancy and small motion.

47:51 Lucas-Kanade Adds Spatial Coherence

The Lucas-Kanade method assumes neighboring pixels move together and share the same optical-flow vector.
Spatial coherence is often reasonable inside a moving object but breaks down at object boundaries where foreground and background motions differ.
A local window, commonly 5×5 or 7×7 pixels, supplies multiple flow constraints for one shared motion estimate.

51:00 Build a Windowed System of Optical-Flow Equations

For a 5×5 window, the method stacks 25 equations of the form Ix u + Iy v = -It.
The shared flow vector has two unknowns, while the 25 pixel constraints make the system overdetermined.
Window size depends on image resolution and must balance the number of constraints against the assumption that nearby pixels share motion.

56:21 Estimate Local Flow with Least Squares

Writing the windowed constraints as A d = b, with d = [u, v]ᵀ, gives the least-squares estimate d = (AᵀA)⁻¹Aᵀb when the inverse exists.
Repeating the calculation around each image location produces an optical-flow field.
The 2×2 matrix AᵀA is invertible only when the window's image gradients provide enough independent directional information.

1:01:02 Textureless Patches, Edges, and Limits of Flow-Based Localization

A textureless patch has near-zero spatial gradients, so motion cannot be inferred from its image intensities.
A single edge constrains motion across the edge but leaves motion along the edge ambiguous, making AᵀA singular or poorly conditioned.
Lucas-Kanade can produce jitter and inconsistent vectors; learned optical-flow models may improve accuracy but still have errors.
Integrating an imperfect velocity estimate to recover position compounds error, motivating sensor fusion and measurement aggregation in later state-estimation lectures.

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Introduction to Robotics @ Princeton.