cs6465 Advanced OS - Lecture 02 Virtualizability of x86 (continued)

Overview

Anton Burtsev examines x86 virtualization problems where sensitive instructions do not trap, focusing on descriptor-table reads and the PUSHF/POPF flag instructions discussed in Robin and Irvine's work. He reviews x86 privilege levels, kernel entry and interrupt handling in xv6, then uses source-code investigation to show why a guest kernel running at ring 3 may observe or change hardware flags differently than it expects.

Key takeaways

A VMM can generally emulate sensitive instructions when they trap, but a sensitive instruction that executes silently can reveal or alter host hardware state without giving the VMM control.
x86 privilege transitions rely on hardware-managed state: an interrupt uses a ring-0 stack from the TSS, and IRET restores a trap frame to return to ring 3.
A separate kernel stack is essential because user code controls its stack pointer and the CPU must save interrupt state before kernel code can inspect that pointer.
CPL, DPL, and RPL describe different parts of x86 protection: the active CPU level, a descriptor's privilege, and a requested privilege constraint.
PUSHF can expose hardware RFLAGS to a ring-3 guest, while POPF at ring 3 can silently ignore changes to privileged bits such as IF, creating a mismatch between guest expectations and actual CPU state.
xv6's readflags implementation provides a concrete source-code path for investigating the virtualization hazard: it uses PUSHF and POP to read flags, and initialization checks whether interrupts are enabled.

Chapters

0:00 Course Updates and the x86 Virtualization Question

Anton Burtsev says the first paper's GradeScope questions have been reopened for late submissions.
The lecture continues examining instructions that are sensitive but do not trap, a central issue in x86 virtualization.

2:16 Why Non-Trapping GDT Reads Complicate Virtualization

Instructions such as store-GDT can expose the address of a guest's global descriptor table without trapping.
A virtual machine monitor may relocate the guest GDT, so software that reads and then directly updates the returned address can encounter a different or inaccessible structure.
Such behavior may not break a practical kernel like xv6, but it can matter to kernels that verify descriptor-table addresses or to software detecting virtualization.

6:14 PUSHF, POPF, and the Need to Understand CPL

PUSHF saves EFLAGS on the stack without trapping, while POPF restores flags subject to privilege restrictions.
Burtsev introduces CPL, DPL, and RPL because POPF behavior depends on the current privilege level.
The class begins investigating whether these flag instructions can break an x86 guest.

10:48 How the GDT Sets x86 Privilege Levels

In segmented x86, each GDT descriptor can specify a base, limit, access properties, and descriptor privilege level from 0 to 3.
Loading a code-segment selector such as CS causes the CPU to consult the GDT; the selected descriptor determines the current privilege level.
A typical kernel uses ring 0 code and data descriptors, while user code and data use ring 3.

18:15 Finding the Kernel-to-User Transition in xv6

Burtsev assigns students a source-code investigation: find how x86 moves from kernel ring 0 to user ring 3.
He recommends semantic navigation through familiar process-creation code, documentation, and IDE cross-references rather than relying only on blind text searches.
The assignment uses xv6 to connect the privilege-level explanation to real kernel implementation.

23:02 Tracing xv6 from the Boot Entry Point to main

The xv6 kernel linker script resolves the entry point to the assembly label entry in entry.S.
The boot path sets up execution state and the stack before transferring control to main.
Burtsev recommends using the xv6 book, annotated source repositories, and earlier systems-boot lectures to understand the code.

30:55 Kernel Initialization Creates the First User Process

xv6 main initializes memory allocation, interrupt controllers, segment descriptors, the console, the filesystem, and other kernel subsystems.
The process-init path configures user code and data descriptors with privilege level 3.
The initialization sequence leads to the trap-frame setup used to enter user mode.

34:46 Why x86 Needs a Separate Kernel Stack

A user process controls its own stack pointer, so the kernel cannot safely use that pointer when handling an interrupt.
If the CPU pushed interrupt state onto an attacker-controlled pointer into kernel memory, it could overwrite kernel data.
A user attempt to access protected kernel memory normally faults, but interrupt entry must save state before kernel code can validate the user stack.

43:04 Interrupt Entry Uses the TSS to Find a Trusted Stack

An interrupt descriptor supplies a handler address and a kernel code-segment selector, allowing a transition from CPL 3 to CPL 0.
On a privilege transition, x86 obtains the ring-0 stack from the Task State Segment, or TSS.
The hardware saves the user stack state and other return information on the trusted kernel stack before executing the handler.

48:11 Trap Frames and IRET Restore User Privilege

The interrupt transition saves the prior code segment, instruction pointer, flags, stack segment, and stack pointer in the trap-frame state.
IRET restores those values from the stack and returns execution to the saved user instruction.
The saved user code-segment selector points to a ring-3 GDT descriptor, so IRET changes the CPU back to CPL 3.

52:00 CPL, DPL, and the Historical Role of RPL

CPL is the CPU's current privilege level, while DPL is a privilege field in a descriptor.
RPL, the requested privilege level, is an older segmentation mechanism that can constrain access using a caller's requested privilege.
Burtsev explains why Robin and Irvine's discussion of RPL can be difficult to apply to modern systems, which rarely use the feature.

58:30 POPF Silently Restricts Flag Changes at Ring 3

At ring 0, POPF can restore the flags under kernel control; at ring 3, the CPU prevents changes to privileged flag bits such as the interrupt flag.
The instruction's behavior changes with privilege level even though it does not trap, making it sensitive under the virtualization classification.
A guest kernel running in ring 3 may therefore fail to get the flag state it requested.

1:00:38 Trapped Instructions Can Be Emulated, Silent Ones Cannot

Burtsev walks through xv6's entry assembly and examples involving CR4, CR3, and model-specific registers.
When a privileged instruction traps, a VMM can decode it, emulate the guest-visible effect, update guest state, and resume at the next instruction.
The contrast is that PUSHF and POPF do not trap, so a VMM may not get a chance to provide guest-specific flag behavior.

1:07:21 A Ring-3 Guest Shares Hardware Flags with the Host

The proposed setup runs a host kernel at ring 0 and a virtualized x86 guest kernel as a ring-3 process, with guest processes also running in ring 3.
Trapped guest instructions can be handled by the host VMM, but PUSHF can read the actual hardware RFLAGS value directly into guest memory.
POPF may then silently refuse to update the interrupt flag, leaving the guest's intended flags inconsistent with the hardware state.

1:10:00 xv6 readflags Exposes the PUSHF/POPF Investigation

Students locate xv6's readflags routine, which uses PUSHF followed by POP to copy the flags into a variable.
The lecture connects that implementation to mycpu initialization, which checks the interrupt-enable flag and can panic if it is unexpectedly set.
Burtsev leaves the exact guest failure case for further investigation, asking students to determine how silent flag behavior affects xv6 under virtualization.

Transcript excerpt

0:35 Okay, so ready to go. So, I don't have any bad news for today, which is great. Uh So, which is just amazing. Nobody quit. Uh uh Everything is on track. Uh um Um so, I reopened uh the late uh the the first uh the grade the gradebook assignment, which I still which I still need you to answer the questions about the first paper. Uh I think it's open until Friday. So, like if you or maybe until 18 today, I forgot it already. I can I can double check. But uh just if you if you submitted late on gradebook, you submit there. And I will assign questions about

1:25 uh Irwin and Robin. But I just wanted to like talk through all of it uh one more time with you because it got like uh it actually got a little a little weird to some degree. Uh even for me, honestly. Like uh it looks it looks very simple on the surface, but then when you start digging, it gets uh it gets a little surprising. Uh That's what I wanted to say. And uh the plan today is just like uh let me give you um like let's go through through go through all these instructions and and I can send the general arch and then try to get on our laptops and try to find those instructions and next few days. And this is where it gets interesting a

2:12 little and then discuss what we got out of it. So we kind of stopped roughly here last last time, right? So essentially we were talking about all of those different cases of instructions which are non-virtualizable because they're sensitive, they don't trigger an interrupt or a trap, sorry. And so essentially at a high level

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.