Princeton Intro to Robotics (Fall 2026) | Lecture 14: Visual SLAM
Overview
The lecture connects occupancy-grid SLAM to visual SLAM, where RGB images alone are used to estimate camera poses, depth, 3D point maps, and cross-frame features. It explains VGGT (Visual Geometry Grounded Transformer) as a learned alternative to classical bundle adjustment, then develops bundle adjustment’s reprojection-error objective and the practical problems of calibration, mismatched correspondences, outliers, initialization, and scale ambiguity.
Key takeaways
- VGGT takes an arbitrary sequence of RGB images without camera calibration and predicts camera intrinsics and extrinsics, depth maps, point maps, and features.
- A point map is not just a depth image: converting depth into world-frame XYZ coordinates requires camera geometry, including intrinsics and pose.
- Bundle adjustment jointly estimates 3D landmarks and camera poses by minimizing the difference between observed image locations and their projected locations.
- Classical visual SLAM is vulnerable to incorrect feature matches, poor initialization, calibration errors, and insufficiently informative views; RANSAC, Huber loss, and key-frame selection mitigate but do not remove these issues.
- Monocular geometry has inherent scale ambiguity, while learned scene priors let VGGT estimate plausible metric distances; atypical scenes can undermine those estimates.
- VGGT’s demonstrations show rapid reconstruction at several scales, from a single image to 128 views, but learned predictions rely on prior data rather than explicitly solving bundle adjustment.
Chapters
- Occupancy-grid mapping discretizes space and stores each cell’s probability of being occupied, from 0 to 1.
- A probabilistic range-sensor model accounts for missed obstacles, false detections, and noisy distance readings.
- SLAM jointly estimates the robot’s state and the map; knowing either one exactly reduces the task to localization or mapping.
- The lecture’s visual-SLAM setup assumes RGB cameras only, without a depth camera or range sensor.
- The goal is to establish the problem formulation and bundle adjustment before examining the learned pipeline in the next lecture.
- Classical visual SLAM explicitly optimized geometric consistency, while newer approaches learn a direct mapping from images to scene geometry.
- VGGT, short for Visual Geometry Grounded Transformer, turns RGB views into a 3D point cloud.
- Demonstrated examples used 32 images in about half a second, 64 images in roughly 1.5 seconds, and 128 images in about 4 seconds.
- The examples also show two-view reconstruction and single-image geometry, though the two-view result has a visible stitching gap.
- The input is a sequence of N RGB images, ranging in the examples from a single image to more than 100.
- VGGT does not require pre-supplied camera calibration, focal length, or field of view.
- Its four outputs are camera intrinsics and extrinsics, a depth map, a point map, and learned features for correspondence and tracking.
- A depth map assigns each pixel a scalar distance to the nearest visible scene point along that pixel’s viewing ray.
- A point map assigns each pixel an XYZ coordinate in a shared world frame, which VGGT anchors to the first camera’s frame.
- Depth alone is insufficient to recover XYZ coordinates without camera information such as the pinhole model’s focal length.
- Extrinsics describe each camera’s position and orientation, while intrinsics encode image geometry such as horizontal and vertical field of view.
- Each image receives a feature tensor with height, width, and C feature channels.
- The same physical point should have similar feature vectors across views, even when its pixel location changes.
- Cross-image feature matching can support point tracking and optical-flow estimation, although VGGT is heavyweight for optical flow alone.
- A classical pipeline calibrates pinhole-camera intrinsics using a checkerboard with known physical dimensions and images taken from multiple poses.
- Feature correspondence identifies image pixels that depict the same 3D point across different views.
- Traditional pipelines used handcrafted SIFT features or learned features to establish those matches.
- For each matched point, project its estimated 3D position into each camera and compare the prediction with its observed pixel location.
- Bundle adjustment jointly estimates 3D point positions and camera extrinsics to minimize the summed reprojection error.
- The first camera’s coordinate frame fixes the otherwise arbitrary global translation and rotation of the reconstruction.
- A visibility indicator sets the reprojection-error weight to zero when a point is not visible in a particular image.
- Incorrect feature matches create large residuals; RANSAC can reject outliers, while a Huber loss reduces their influence.
- The Huber loss is quadratic near zero error and becomes linear for large errors, limiting the penalty from bad correspondences.
- Classical solvers such as Levenberg-Marquardt iteratively adjust camera poses and point locations to reduce reprojection error.
- COLMAP is a widely used classical reconstruction pipeline that can produce large, geometrically consistent point clouds from many photographs.
- Optimization quality depends on a good initial estimate, since a poor initialization can lead to an incorrect local minimum.
- Calibration errors, weakly textured scenes, and mistaken feature correspondences can cause classical reconstructions to fail or collapse into distorted point clouds.
- Key-frame selection avoids redundant near-identical views, while outlier rejection helps but does not eliminate incorrect matches.
- Sequential systems can struggle to recover when earlier pose or map errors become the initialization for later frames.
- Bundle adjustment has scale ambiguity: scaling the scene and camera translations can leave reprojections unchanged.
- VGGT can estimate plausible absolute distances by using learned priors about familiar objects and environments, despite the geometric ambiguity.
- Unusual scenes such as dollhouses or optical-illusion rooms can mislead those priors; even a small amount of depth information can resolve scale ambiguity.
- The midterm is scheduled for Tuesday, November 10, during lecture time, and is designed to be completed by hand without electronic devices.
- Exam coverage ends with the Thursday lecture on SLAM; a TA-led review is planned for the following Thursday.
- Students reported trying multiple drones before finding a working unit; labeling each drone with its last-tested date was suggested to improve handoffs.
Transcript excerpt
0:10 All right, I think we'll get started. Yeah, I guess welcome back from from break. Hopefully you had at least a few relaxing days. So the plan for today is to continue with our discussion of localization and mapping, specifically slam, so simultaneous localization and mapping. So in the last couple of lectures, we saw an application of particle filtering in particular, which is itself an instantiation of the the base filter to this problem of slam. So simultaneously localizing your robot in some environment and then figuring out what the environment looks like. And I want to just give give you a quick recap since since it's been a a few days. So the main kind of method for slam that we discussed was what's known as occupancy grid mapping. So this is an example of a drone that's
0:57 flying around in this environment. This left top left view is a first person view. So RGB camera views from the drone. This is a third person view. And then this is the the map that the drone is building as it's going along. So as it's going along, it's collecting sensor information from on board sensors. So cameras like RGB cameras, depth cameras. And it's building this map. So it's localizing itself. So it's figuring out where in the environment the the drone is. And then it's figuring out what the the map looks like. And specifically what we discussed was occupancy grid mapping. So in occupancy grid mapping, you have some discrete representation of the environment. So you discretize your your space.
1:45 And then you basically just represent for each cell whether that cell is occupied or not. So it's it's a one if that cell is occupied. It's a zero if that cell is not occupied. And yeah, you basically just
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Introduction to Robotics @ Princeton.