cs6465 Advanced OS - Lecture 09 - Comparison of HW and SW techniques for virtualization

Overview

Anton Burtsev uses the 2005 comparison of software and hardware x86 virtualization to explain binary translation, translation-cache chaining, guest memory isolation, and shadow page tables. The central historical result is that binary translation could outperform early Intel VT-x without Extended Page Tables because shadow-page-table traps were expensive, while EPT later made hardware-assisted virtualization generally faster.

Key takeaways

Binary translation avoids interpreting every x86 instruction by translating basic blocks from the guest program counter, leaving ordinary instructions unchanged and redirecting sensitive or control-flow instructions.
Translation-cache chaining patches known branches to jump directly between cached blocks, so frequently executed paths can run almost entirely in translated code.
Ring compression places the VMM at ring 0, the guest kernel at ring 1, and guest applications at ring 3; segmentation could protect VMM memory on 32-bit x86 but normal segment limits are unavailable in Intel 64-bit mode.
Shadow page tables map guest virtual addresses directly to host physical addresses, and the VMM must synchronize them with guest page-table changes using eager write traps or lazy hidden page faults.
Early VT-x without EPT could lose to binary translation because shadow-page-table maintenance was expensive; EPT and AMD nested paging made hardware-assisted virtualization generally faster.
Binary translation remains useful beyond virtualization, including QEMU's cross-architecture execution and Apple's Rosetta translation of x86 applications to ARM.

Chapters

0:00 Lecture Setup and the 2005 Virtualization Comparison

Anton Burtsev frames the paper as a way to understand software and hardware virtualization techniques, rather than as a comparison that still applies unchanged today.
The class discusses using LLMs for operating-system homework, including a planned paravirtualized x86 exercise without hardware-assisted virtualization.
The lecture builds on Popek and Goldberg's trap-and-emulate framework and Robin and Irvine's analysis of x86.

3:22 Why x86 Needs Binary Translation Instead of Simple Emulation

Popek and Goldberg's criteria include fidelity, performance, and keeping the virtual machine monitor in control.
Robin and Irvine show that x86 has sensitive instructions that do not trap when run outside the most privileged ring, such as reading privilege-related state or using POPF.
A software emulator can fetch, decode, and execute each guest instruction, but may require hundreds of host instructions to emulate one guest instruction.
Binary translation instead rewrites selected instructions and runs most translated guest code directly on the host processor.

10:00 The Prime-Number Example and Basic Translation Blocks

The paper illustrates translation with an is-prime routine testing 49 by repeatedly dividing it and checking remainders.
The translator starts at the guest program counter instead of trying to identify all code in advance, which avoids guessing in the presence of code and data overlap or self-modifying code.
It translates a bounded instruction sequence, such as up to roughly 12 instructions, or stops at a control-flow change to form a basic block.
Ordinary register operations can often remain identical, while control-flow instructions and sensitive operations need adjustment.

13:09 Branch Patching and Chaining Translated Blocks

A translated branch initially exits to the translation engine if its destination block has not yet been translated.
When execution reaches a fall-through block, the translator can place its translated code next to the prior block and eliminate the original branch.
Once a branch target has a cached translation, the translator can patch the branch to jump directly to that cached block.
Chaining lets frequently used paths loop within the translation cache instead of repeatedly entering the compiler.

18:24 Dynamic Returns and Translation-Cache Lookup

A return instruction has a destination taken from the guest stack, so the target cannot always be patched as a fixed address.
The translated return places the guest return address in a register and dispatches through the VMM using GS-relative state.
The VMM uses the address to search a hash table for a translation-cache entry, jumping to it if present or translating it on demand.
The extra sequence preserves VMM control instead of blindly transferring to an address supplied by guest code.

27:53 Which x86 Instructions Can Stay Identical

General-purpose register operations can usually execute unchanged, while privileged operations involving control registers such as CR3 must be emulated or redirected.
Fall-through execution can remain direct, and a static branch can be patched once its target is in the translation cache.
Indirect branches through registers require runtime lookup because their destinations can vary between executions.
PC-relative loads, calls, and jumps need adjustment because translated blocks occupy different addresses and layouts from the guest code.

32:10 Protecting VMM Memory from Translated Guest Code

A guest instruction such as MOV RBX, [RAX] can access an address chosen at runtime, so the VMM must prevent guest memory operations from reaching protected VMM memory.
Software fault isolation can insert address checks or masking, but added instructions create overhead; the lecture notes masking strategies used by systems such as Native Client and WebAssembly.
A single shared page table cannot by itself distinguish guest-kernel access from VMM access if both execute with supervisor permissions.
Separate address spaces can enforce isolation, but switching page tables complicates ordinary kernel access to user buffers.

39:38 Ring Compression and Segmentation Limits

A classic ring-compression arrangement places the VMM at ring 0, the guest kernel at ring 1, and guest user processes at ring 3.
On 32-bit x86, segment limits can stop guest-kernel memory references from reaching the VMM even when both regions appear in the same page table.
Intel 64-bit mode largely removed normal segment-limit checks, making this protection scheme unavailable there; the lecture contrasts it with AMD compatibility support.
Without usable segmentation, alternatives include software fault isolation or separate address spaces with page-table switching.

44:00 Guest User Code, Virtualization Leaks, and Ring Behavior

Most guest user code can run without binary translation because it already executes at ring 3 and makes system calls into the guest kernel.
Unprivileged instructions can still reveal virtualization details, for example by storing descriptor-table state and observing addresses that differ from a bare-metal system.
Timing can also reveal virtualization, so the design accepts that a determined guest may detect the VMM rather than translating all user code.
VMware products and QEMU used binary-translation techniques, while QEMU also supports cross-architecture emulation such as running x86 code on ARM.

51:11 Guest Physical Addresses and Shadow Page Tables

A virtualized system has guest virtual addresses, guest physical addresses, and host physical addresses.
A guest cannot safely load its own page-table root directly into the real CR3, because it could map VMM memory.
Before hardware nested paging, the VMM built a shadow page table mapping guest virtual addresses directly to host physical addresses.
The guest page table remains the guest's apparent view, while the hardware uses the VMM-controlled shadow table, which must be kept consistent.

53:54 Eager Shadow-Page-Table Synchronization

In the eager approach, the VMM marks guest page-table pages read-only so writes to page-table entries cause traps.
When the guest loads a new page-table root or adds lower-level entries, the VMM validates the guest-physical mapping and updates the shadow table.
Removing an entry also traps, allowing the VMM to remove the corresponding shadow mapping and flush relevant TLB state.
Eager synchronization is functionally straightforward but incurs overhead on page-table writes.

59:33 Lazy Shadow Tables and Hidden Page Faults

In the lazy approach, the guest can write its page tables normally, and the VMM builds shadow mappings only when guest execution first uses them.
A missing shadow mapping triggers a hidden page fault; the VMM walks the guest page table, validates the entry, and adds it to the shadow table if valid.
If the guest address is genuinely unmapped, the VMM injects the page fault into the guest instead of resolving it silently.
Stale shadow entries require invalidation when the guest changes mappings; intercepted TLB-flush instructions can remove entries, while handling CR3 reloads raises additional synchronization questions.

1:04:58 Why EPT Changed the Hardware-versus-Software Result

The first Intel VT-x systems lacked Extended Page Tables, so both hardware-assisted VMs and binary-translation systems relied on shadow page tables.
Shadow-page-table traps could make early hardware virtualization slower than binary translation, whether synchronization was eager or lazy.
Binary translation improved page-table handling by profiling frequent guest writes and rewriting selected MOV instructions to enter the VMM directly instead of faulting.
Once Intel EPT and AMD nested paging became available, hardware-assisted virtualization generally became faster than binary translation.

1:08:29 Modern Uses of Binary Translation and Cross-Architecture Emulation

Hardware-assisted virtualization is now dominant, while binary translation remains useful for niche cases such as running x86 binaries on ARM.
QEMU can dynamically translate x86 instructions on ARM, but that does not necessarily provide complete VMX emulation or nested virtualization.
Apple Rosetta is discussed as a mature production binary-translation system, though the lecture notes that unusual workloads can still expose compatibility failures.
The enduring lessons are how translation caches, shadow page tables, and hardware-assisted virtualization are constructed, rather than the paper's dated benchmark rankings.

Transcript excerpt

0:15 Okay, so welcome back everyone. Let me quickly share the screens. I'm sharing this one. And this one too. So, the idea is to today is to discuss this paper which compares hardware and software techniques for x86 virtualization. But really is to and comparing those techniques was meaningful in 2005, right? Now this comparison is different, but it's a reasonable paper to understand how binary translation or software-based techniques for virtualization work, right? And there's another paper which

1:03 written by VMware guys about VMware. It's so nuanced and so low low-level that maybe it's not worth digging for, but maybe maybe we'll take a look later. But this is probably good enough for you to understand what it is. Uh so it's less of a comparing those techniques, more about understanding those. Uh but we let me come back to the paper or to the So, it's the very first time we're doing those homeworks. AI took over, you know, I didn't know how to properly frame this. So, and my my idea was again, I will assign it because in the past I like those blog post-like build your own operating system from scratch style things. And this is what I

1:52 did for this homework one, and this is what I planning on doing for homework two. But you tell me whether it was uh fun or you prefer a different format. Because it was a lot of reading and a lot of coping and maybe not so much probably a ton of reading if you read it all. But actually I read it. I didn't write it but I read it, right? Uh I used a LLM to write it mostly and let's talk to a LLM. Uh but this is a case this is this works

This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.

In this course

Related notes

Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.