cs6465 Advanced OS - Lecture 02 - Virtualizability of x86 (part 3)

Overview

Anton Burtsev analyzes why xv6 is not directly virtualizable on classic x86, using the Popek-Goldberg sensitive-instruction problem and a ring-compression design like Xen's. The lecture traces failures involving pushf/popf, CLI/STI, shadow RFLAGS, GDT and IDT state, and ring-1 execution, then examines interrupt injection and an unresolved IRET corner case before assigning Intel VT-x reading.

Key takeaways

xv6's pushcli and popcli rely on pushf to inspect RFLAGS, but pushf does not trap at guest CPL 1, so those routines can read the hypervisor's physical IF state instead of the guest's shadow IF.
A hypervisor cannot safely implement a guest CLI by clearing physical IF; it can keep host interrupts enabled and emulate the guest's disabled state in shadow virtual-CPU data.
Ring compression maps a guest kernel's nominal CPL 0 to actual CPL 1, preserving CPL 0 for the hypervisor, but breaks guest code that assumes kernel CS has privilege bits equal to zero.
Guest GDT, IDT, and TSS state must be shadowed because descriptor limits, interrupt destinations, and stack-switch pointers directly affect isolation and control flow.
Software interrupt injection requires reconstructing the guest's interrupt frame and using shadow IDT and TSS data; IRET's restoration of guest flags remains an unresolved corner case in this lecture.

Chapters

0:00 Why xv6 pushcli and popcli Fail Under x86 Virtualization

The opening question concerns xv6's pushcli and popcli routines, which coordinate interrupt disabling around critical sections.
CLI traps when the guest kernel runs below the required privilege, but pushf reads hardware RFLAGS without trapping.
That asymmetry can make xv6 inspect host interrupt state instead of the guest's virtual interrupt state.

5:00 How pushf, popf, and IOPL Affect Guest Flags

pushf places the actual hardware flags on the stack, exposing state that belongs to the host or hypervisor.
At CPL 0, popf can update flags; at a lower privilege such as CPL 1, sensitive fields including IF may be ignored.
IOPL in RFLAGS controls whether CLI, STI, and IN/OUT instructions can execute without trapping.

12:00 Shadowing xv6's Interrupt-Enable State

A hypervisor cannot simply clear the physical IF bit for a guest, because doing so can prevent it from regaining control through maskable interrupts.
Instead, the monitor can leave hardware interrupts enabled and record the guest's requested IF value in shadow virtual-CPU state.
xv6's pushcli reads real RFLAGS, however, and popcli checks that value, so the mismatch can trigger a panic after the first interrupt-disable sequence.

20:00 Searching xv6 for Sensitive x86 Instructions

Burtsev recommends searching the xv6 source for instructions identified as sensitive in Robin and Irvine's analysis.
The useful search set is narrowed to instructions xv6 actually uses, rather than every problematic x86 instruction.
The lecture notes that xv6 accesses control registers and descriptor tables, while several other candidate instructions do not appear in its code.

23:30 xv6's Far Jump and the Global Descriptor Table

During boot, xv6 uses a far jump to select a code-segment descriptor after configuring the GDT.
A GDT entry includes a privilege level, base, and limit, so allowing a guest to modify the real table could undermine isolation.
The segmentation limit matters for separating a guest kernel from a hypervisor when the guest runs at CPL 1.

35:00 Shadow GDTs and Ring Compression in Xen

The hypervisor maintains a shadow GDT instead of loading the guest's writable GDT directly into hardware.
When the guest loads its GDT, the monitor copies and adjusts descriptors, mapping the guest's kernel privilege level 0 to actual CPL 1.
This ring-compression scheme preserves guest user execution at CPL 3 while reserving CPL 0 for the hypervisor, an approach used by Xen on 32-bit x86.

39:00 Finding xv6 Code That Assumes Kernel CPL 0

The class searches xv6 for uses of CS that could reveal the guest kernel's actual CPL 1.
The trap-handling path classifies faults using the low bits of the saved CS selector and assumes kernel execution is at privilege level 0.
Under ring compression, a CPL 1 fault can be misclassified, causing xv6 to handle a kernel fault as though it came from a user process.

45:00 How IDT Entries Use GDT Code Segments

An IDT entry's code-segment selector identifies a descriptor in the GDT, which determines the privilege level used for interrupt handling.
The IDT descriptor privilege level controls whether software running at CPL 3 may invoke an interrupt gate, as with a system-call vector.
Hardware timer vectors normally use a more privileged gate, so user code cannot forge a timer interrupt with an INT instruction.

51:00 Why xv6's Trap Handler Misreads Ring-1 Faults

xv6's trap path distinguishes user faults from kernel faults by examining the saved CS privilege bits.
The kernel's checks assume its code segment has CPL 0; a compressed guest kernel instead has CPL 1.
As a result, fault handling can take the wrong branch and continue with invalid assumptions about the process or kernel state.

56:00 Shadowing the Task State Segment and Kernel Stack

The guest must not control the hardware TSS, because its privilege-transition stack pointers affect interrupt entry.
A hypervisor shadowing the GDT, IDT, and TSS must recreate their relationships, including redirecting the guest's kernel stack transition from CPL 0 to CPL 1.
Incorrect TSS shadow state can make hardware enter the guest kernel on the wrong stack.

58:00 Routing Timer Interrupts Through a Shadow IDT

The hypervisor can build a shadow IDT that routes selected interrupts, such as the timer, to a monitor handler.
Other entries, including a system-call path, may be configured to transfer control directly to the guest kernel when monitor intervention is unnecessary.
Correct shadow GDT and TSS setup lets hardware perform the privilege transition and stack switch for directly delivered interrupts.

1:05:00 Interrupt Gates, Trap Gates, and IF Behavior

IDT gate configuration determines whether interrupts are disabled on entry; interrupt gates clear IF, while trap gates preserve it.
xv6 can use different gate behavior for system calls and hardware interrupts, affecting whether the kernel may be preempted immediately.
The hypervisor must account for these guest-visible flag semantics while retaining control of physical interrupts.

1:09:00 Injecting a Timer Interrupt into the Guest

To inject a timer event, the monitor can use the guest's shadow IDT to find the vector handler and the guest TSS to locate its kernel stack.
It constructs the interrupt frame, including saved instruction state, CS, RFLAGS, stack pointer, and stack segment, then returns into the guest handler.
This software-created frame makes the guest observe an interrupt transition even though the physical event first entered the hypervisor.

1:13:00 The Unresolved IRET and Guest RFLAGS Corner Case

The lecture examines how IRET restores guest RFLAGS when returning from an injected interrupt, particularly when the guest kernel runs at CPL 1.
Some flag writes may be ignored at that privilege level, creating a possible mismatch between saved guest IF and the resumed execution state.
Burtsev identifies that the slide's proposed failure scenario is not yet convincing and postpones resolving the interrupt-return case.

1:17:00 Preparing for Intel VT-x and the Next Assignment

The next class begins hardware-assisted virtualization with Intel VT-x, which reduces the need to emulate classic ring-compression behavior.
Students are asked to begin reading several substantial Intel developer-manual chapters before the lecture.
The discussion emphasizes that x86 virtualization has many corner cases and that the current interrupt-return analysis still needs verification.

Transcript excerpt

0:29 Uh okay. So well let's try to answer the question about uh push f which I was asking uh online. So who has a version for us? What? Uh remember >> homework told you only later today. >> But did you guys did you have a chance to take a look? Okay, tell me what will go wrong. Here's my X6 source code. So, how many how many people have an answer or close to an answer? Two. Okay, guys. Like, okay, let me ask differently. Why do you think you don't have an answer? the question is uh clear or not but

1:19 still no like uh we if I remember correctly what we were looking at uh I hope it's somewhere here in the like okay sure search here my CPU we search for something like choir this um no like we looked at this source code right and uh this push CLI if we go into the push CLI we see that it uh uh reads the flags uh does actually CLI and stuff like that but still you wait the goal

2:10 here I again I did talk a lot, right? It's not going to help you. The only thing which will help you is if you start thinking then then it will help the only way to get to win this game. I I'm here only to help you guys. >> Okay, let's hear the answer or close to an answer. >> So I mean as far as I can tell F never is used. >> Push F is used for I think it's used in the spin >> uh >> um break if the host like enables or disables interrupts then the interrupt flag going to read the interrupt flag from the host OS which >> no

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.