cs6465 Advanced OS - Lecture 05 - Designing an xv6 VMM

Overview

Anton Burtsev guides the class through a low-level design for an xv6-hosted, type-2 x86 VMM using Intel VMX, with a small kernel component and a user process that creates and runs a guest. The initial milestone is a guest “hello world”: allocate a page-aligned VMCS, set up EPT mappings and guest state, load a binary image, handle VM exits and interrupts, and print through serial I/O before adding a cleaner hypercall interface.

Key takeaways

A minimal xv6 VMM can keep privileged VMX operations in the kernel while using a normal user process to request VM creation and execution, following a type-2 design inspired by KVM.
VMCS setup and VMX enablement are CPU-specific: VMX must be enabled on the CPU that runs the guest, and the VMCS needs a page-aligned region.
Guest virtual-to-physical translation and EPT translation solve different problems: guest page tables map guest virtual addresses to guest physical addresses, while EPT maps guest physical addresses to host physical memory.
A small flat guest binary linked to a fixed address, such as 1 MB, avoids implementing a full executable loader for the first milestone, but the image still must be copied into EPT-backed host pages.
VM exits let the host handle interrupts and guest requests; direct serial-port I/O is a simple first output mechanism, while a trapped instruction such as `CPUID` can serve as a hypercall interface.
The first milestone is intentionally not a full guest OS: it is a VMX guest that can run a small binary, print output, handle exits, and clean up its EPT-backed memory.

Chapters

0:00 Designing an xv6 VMM as a Class Exercise

The class shifts from the planned material to sketching an x86 VMM design as an in-class activity.
Students are asked to describe the design at a reasonably low level, including where the VMM runs, required processes, and data structures.
The design exercise will become graded homework, with a plan due by the end of class.

7:00 Framing the VMX Hypervisor Design Problem

The exercise focuses on building an x86-64, type-2 hypervisor using hardware virtualization.
Students first spend time developing a design outline before discussing its components together.
The goal is a practical plan detailed enough to guide implementation or explain the design to colleagues.

17:00 Choosing a Type-2 Design Inspired by KVM

Burtsev favors a type-2 design, while allowing students to consider type 1 as an alternative.
KVM is presented as a hybrid model: a small amount of functionality runs in the kernel, while more is kept outside it.
The class starts with a working design and can later move components behind cleaner interfaces for modularity.

20:30 Why Intel VMX Avoids Instruction Emulation

The target is an Intel VMX-based hypervisor, which relies on CPU hardware virtualization.
Without VMX, a VMM would need to scan guest instruction streams and potentially use techniques such as JIT compilation.
The first design should establish the overall VMX execution path before detailing every instruction trap.

35:00 Enabling VMX on xv6 Host CPUs

VMX is enabled in the host's privileged execution context, not by an ordinary guest or ring-3 program.
The class considers enabling VMX once per CPU during xv6 startup, around `mpmain`, rather than only once globally.
Enabling VMX puts the host into VMX root operation, which is distinct from the guest's VMX non-root operation.

38:00 Starting a VMM Process and Allocating a VMCS

Rather than turning the shell or the entire kernel into the VMM, xv6 can launch a regular user process such as `runvm`.
That process enters the kernel through a system call to create a VM and prepare its state.
The kernel can allocate a page for the VMCS using xv6's page allocator, which provides page-aligned memory suitable for the VMCS region.

42:00 Configuring VMX State for the First Guest

The kernel must validate the VMCS region and configure the control and guest-state fields before entry.
VMXON and VMCS setup must occur on the CPU that will run the guest, so the design places the relevant operation in the execution path that invokes VMX.
The first target is deliberately small: configure enough guest state to enter a simple binary, not yet a complete xv6 guest.

45:00 Defining the Minimal Guest and VM-Entry Loop

The first guest can be a tiny binary that prints “hello world,” with booting a full xv6 kernel deferred to a later stage.
The VMM's basic control loop uses VM entry, handles VM exits, and resumes the guest when appropriate.
A guest launch is a privileged operation, so the xv6 user process must request it through the kernel.

48:00 Using System Calls to Create and Run a VM

The proposed interface separates VM creation from VM execution, using calls such as `sys_create_vm` and `sys_vm_run`.
`sys_create_vm` prepares VM resources; `sys_vm_run` enters the guest with VM launch or resumes it with VM resume.
The regular xv6 scheduler can continue scheduling the `runvm` process, allowing its next VM-run call to resume guest execution without initially replacing the scheduler.

51:00 Setting Guest RIP and Supplying a Guest Image

Before VM entry, the VMCS guest instruction pointer, RIP, must point to the binary's entry point.
The guest program needs to be supplied to VM creation, along with a size describing the guest memory region.
The initial image is a small, self-contained binary; the later goal is to boot an xv6 kernel as the guest.

55:00 Separating Guest Page Tables from EPT

Guest page tables translate guest virtual addresses to guest physical addresses, while Extended Page Tables translate guest physical addresses to host physical addresses.
The proposed guest can begin in 64-bit mode with paging initially disabled, allowing its own boot code to build guest page tables.
The VMM still needs to create EPT mappings so guest physical memory is backed by host physical pages.

1:00:00 Building EPT with xv6 Page Allocation

The VMM can allocate pages and build EPT structures with a page-by-page mapping routine similar to xv6's `mappages`.
The initial EPT maps a guest physical range from zero to the requested VM size onto allocated host pages.
The VM's process data structure should retain an EPT pointer, and the VMCS must be configured to use that EPT.

1:04:00 Loading a Guest Binary from the xv6 Filesystem

For the first small guest, the image can be stored as a file in the xv6 filesystem and passed into `sys_create_vm`.
A linker script can place the binary at a chosen guest physical address, such as 1 MB, avoiding a more complex relocation path.
The class notes that xv6's filesystem has a small maximum file size, around 16 KB in the discussed setup, so a full guest kernel may require changes.

1:08:00 Copying Guest Image Pages into EPT-Backed Memory

The guest image must be copied into the host pages that back its guest physical address range, not merely retained as an xv6 file buffer.
Loading the image requires finding the EPT-backed host page for each guest page and copying the corresponding bytes.
With paging disabled initially, the guest can execute at its configured address through segmentation and EPT, then enable its own page tables later.

1:11:00 Handling Guest Interrupts and Resuming Execution

The VMCS should be configured to exit on interrupts so host events, including timer interrupts, can preempt guest execution.
After a VM exit, the host can enable interrupts and let xv6's scheduler run; a later `sys_vm_run` can resume the guest.
For the initial guest, exiting on exceptions as well helps diagnose failures, while interrupt injection into the guest can wait until it has an interrupt table.

1:14:00 Printing Guest Output and Completing the First Milestone

The simplest initial output path is to let the guest write characters directly to the serial device using x86 `in` and `out` instructions.
A hypercall can provide a cleaner interface: a guest instruction such as `CPUID` triggers a VM exit, and the guest passes arguments in registers for the VMM to process.
When the guest signals completion, the VMM can free EPT resources, return from `sys_vm_run`, and let the xv6 process exit.
Burtsev closes by announcing Zoom lectures for the next two weeks, homework support through office hours, and a target to finish before spring break.

Transcript excerpt

0:37 Okay. So, uh, so today's plan looking at me. It's a little weird. I agree. getting Visual Studio. Yeah, that's today's plan is to take a little break in lectures because we can finish the the MX chapter later but try to design an X6 take it to an exercise. I will I will create the assignment like in class activity scope I will ask you to like to submit a plan but for the first like

1:24 let's take say take 10 minutes just think about and then we'll do step by step and then we'll turn it turn this into our homework I mean I could have assigned it but I think that's it would just be read through this like do whatever You have to do the submit and that's about it. So like get your laptops and start thinking what does it mean to you? We know everything. We we we know almost everything what we need to know to go to host. Maybe we're like and we can start for sure and then maybe like finish with one more half lecture, but I'm I'm not even sure that we need it, right? So maybe maybe we know everything. So think about it for like for five minutes

2:12 like try to take notes and then then we'll discuss the plan and um I mean to be specific I want you to be like reasonably low level you say look this is this is where where the VMM is running this is if we need a process this is a process this is how we can figure all this necessary data structures we max CMCS how we started how we receive stuff like that meanwhile I'll do like an assignment for

3:00 grade Oh, I'm not sharing the screen. Sure.

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.