cs6465 Advanced OS - Lecture 03 - Intel VMX
Watch on YouTube →
Overview
Anton Burtsev explains Intel VMX as a hardware interface for running guest operating systems in VMX non-root mode while a hypervisor controls execution in VMX root mode. He details VMX initialization, VMCS lifecycle and fields, exit behavior, and the state software must save itself, emphasizing that VMCS does not preserve general-purpose registers and that host state is only partially restored.
Key takeaways
- Intel VMX lets guest operating systems run mostly natively in non-root mode, while VMCS execution controls select which sensitive operations and events return control to the root-mode hypervisor.
- VMX initialization is processor-specific: every logical processor needs a VMXON region, and migrating a VMCS requires clearing and reloading it before launching it on the destination processor.
- VMCS is accessed through VMREAD and VMWRITE field encodings, not fixed memory offsets, and its fields divide guest state, host state, execution controls, and exit diagnostics.
- VM entry and exit do not automatically preserve general-purpose or extended registers; hypervisor code must explicitly save and restore the state its handlers or guest execution require.
- Host VM-exit state is intentionally partial: VMCS supplies essential values such as CR3, RSP, and RIP, then the hypervisor continues execution through a configured trampoline.
- The IA32_FEATURE_CONTROL lock and measured boot support protection against an attacker inserting an untrusted hypervisor beneath a running operating system.
Chapters
- The lecture focuses on Intel VMX data structures and execution modes needed to build a hypervisor for running x86 on x86.
- Intel's chapters 26 and 27 list registers and behavior declaratively, often without explaining design motivations.
- Burtsev recommends reading the manual alongside explanatory notes and using an LLM to investigate unclear design choices.
- VMX root mode hosts the hypervisor, while VMX non-root mode runs the guest operating system under hypervisor control.
- Both modes support familiar x86 privilege rings, descriptor tables, and hardware interrupt handling, so guest code can run mostly natively.
- VMX controls determine which events cause exits; interrupts can be configured broadly, while exception exits can be selected individually.
- Some guest operations, including access to certain model-specific registers, are handled by hardware, trapped for emulation, or unavailable to the guest.
- VMX aims to trap sensitive operations infrequently, letting ordinary events such as division-by-zero exceptions remain inside the guest when configured.
- On an exit, hardware records the reason and selected guest state in the VMCS, loads selected host state, and transfers control to the hypervisor.
- A hypervisor checks VMX support, executes VMXON, configures a VMCS, and launches the guest with VMLAUNCH.
- After the first launch, VMRESUME re-enters that guest following an exit; VMXOFF leaves VMX root mode.
- VMX is designed not to expose a simple guest-status bit, although CPUID exits and timing behavior can reveal virtualization.
- CPUID checks VMX availability, and the IA32_FEATURE_CONTROL MSR governs whether VMX instructions can be enabled.
- Locking VMX feature control is intended to prevent an attacker from inserting a stealth hypervisor beneath an already-running operating system.
- Measured boot records hashes of boot components and the hypervisor, allowing later checks that the expected software chain was loaded.
- Each logical processor, including each hyperthread, needs its own VMXON region, backed by a physical page no larger than 4 KB.
- A logical processor can maintain multiple VMCS regions, but only one VMCS is active on it at a time.
- VMPTRLD selects a VMCS, while VMCLEAR flushes its active state and makes it available for transfer or reuse.
- VMX capability MSRs report supported controls, VMCS revision and size, and other processor-specific features.
- CR0 and CR4 have special guest/host masks and shadow values that constrain which bits the guest can change.
- Guest startup assumes required modes such as protected mode and paging are configured, avoiding the complexity of emulating every historical CPU state.
- The VMCS organizes guest state, host state, execution controls, entry controls, exit controls, and exit information into separate areas.
- Execution controls define which events cause exits, while entry controls can inject a pending interrupt when the guest starts.
- VMCS layout is opaque: software uses VMREAD and VMWRITE with field encodings rather than accessing fields through fixed memory offsets.
- A VMCS becomes current after VMPTRLD, and VMLAUNCH marks it launched so later entries on that processor use VMRESUME.
- Moving a VMCS to another logical processor requires clearing it, loading it there, and performing a new VMLAUNCH.
- The VMCS guest-state area holds execution context such as control registers, instruction and stack pointers, flags, segment state, and descriptor-table bases.
- Segment registers cache descriptor information from the GDT or LDT, so the VMCS stores selectors along with cached bases, limits, and access rights.
- Guest GDTR and IDTR state includes the bases and limits of the global and interrupt descriptor tables.
- Selected MSRs, including those used for fast SYSCALL entry and long-mode configuration, are represented in guest state.
- The VMCS does not save general-purpose registers such as RAX, so the hypervisor must save them on exit and restore guest values before entry.
- Floating-point, MMX, and vector state is also managed by software; XSAVE can preserve extended state when needed.
- Saving only the register state required for a particular exit can avoid unnecessary work, but omitting state the handler uses risks corrupting the guest.
- VMCS activity state can represent normal execution, HLT, shutdown, or waiting for a startup interprocessor interrupt.
- Interruptibility state tracks conditions such as IF being clear, blocking after MOV SS, or blocking subsequent NMIs while an earlier NMI is pending.
- The VMX preemption timer can force an exit after a configured interval even when ordinary timer interrupts are delivered inside the guest.
- On an exit, the processor loads selected host values from the VMCS, including CR3, RSP, and RIP, rather than restoring every host register.
- A hypervisor can set host RIP to the instruction after VMLAUNCH and save its current stack pointer so execution resumes like a function return.
- The host exit handler must save guest general-purpose registers before using them and restore the guest context before VMRESUME.
- VMX constructs host code and data segments with a zero base and maximum limit instead of restoring every segment descriptor from memory.
- Host GDTR and IDTR bases come from VMCS fields, while their limits are synthesized; host flags are also initialized to a defined value.
- The selected host state must still provide valid stack, page-table, and descriptor-table context for the first instruction after an exit.
- A VM-exit trampoline first saves guest general-purpose registers and can use XSAVE for floating-point and vector state.
- The handler then examines VMCS exit information to identify the event and decide whether to emulate it or resume the guest.
- Burtsev closes by recommending iterative manual study, using targeted LLM questions to clarify why Intel saves some state in the VMCS but leaves other state to software.
Transcript excerpt
0:40 So the plan for today and probably Monday uh is to keep working through the details of this PX or VMX interface. I kind of gave a very high level overview of what is going on there last time but really it's not enough to to like build the hypervisor which is our goal right and so today we will take a closer look at data structures VM VMX VMCS uh what's the other one I forgot VMX I think VMX root uh and then essentially like understand it now so we can at least
1:28 write bits of code and reason about the code which I'll give you uh when we want to use this to run x6 on top of x6 right so it's like from from last time I assume that the impression was okay it's relatively simple right so you have this data structure it saves you the state of the guest uh saves the state of the host enters the guest it runs in this VTX environment, exits, you continue and so on, right? In practice, it's a little bit more nuanced. And in fact, it's like it's a lot more nuanced to the point that uh like uh if you think about it, you're trying to virtualize the entire Intel CPU on top of like hardware physical Intel CPU like every bit of knowledge
2:16 which you have and you you have like okay sure segment registers uh global descriptor table interrupt descriptor table page tables okay sure that's that's that's reasonably compact but in reality there's like a ton lot of stuff and you have to reason about all of it and luckily like not not all of it is relevant to us but still we need to understand like enough so we can throw a
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.