cs6465 Advanced OS - Lecture 11 - Xen: paravirtualization (part 2), HW3 design
Watch on YouTube →
Overview
Anton Burtsev concludes Xen paravirtualization by explaining event-channel interrupts, split frontend/backend device drivers, grant-table memory sharing, Xenstore, and the performance and security tradeoffs behind these designs. He then develops the HW3 plan for paravirtualizing xv6, covering privileged operations, guest address spaces and protection, and a software approach to interrupt delivery.
Key takeaways
- Xen event channels replace a full hardware interrupt interface with pending and masking bits in a shared info page, while guests take responsibility for polling masked events.
- Xen’s split netfront/netback design keeps the guest’s Linux networking stack intact and uses grant references plus shared rings to exchange requests with a backend.
- Grant mapping, packet copying, and cache-line bouncing can dominate small-packet networking costs; SR-IOV addresses these limits by assigning NIC virtual functions directly to VMs.
- Keeping physical drivers in Domain 0 reduces the hypervisor’s trusted code base and preserves Linux driver reuse, but a compromised backend can still disrupt services it manages.
- A paravirtualized xv6 design can replace selected privileged operations with hypercalls while representing cheap state, such as interrupt masking, in shared memory.
- Because x86-64 offers no usable ring 1 for the guest kernel, separate guest user and kernel page tables can provide isolation, with host-mediated system-call transitions paying the switching cost.
Chapters
- Xen uses virtual event channels to notify guests about device activity, virtual timers, interprocessor events, and notifications between domains.
- A shared info page lets the hypervisor mark pending events and lets each guest track which channels are masked.
- Unlike VMX interrupt-window exiting, Xen expects the guest to check for additional pending events after handling an interrupt.
- When an event is unmasked, Xen delivers it through an upcall that behaves like a software interrupt and invokes a registered guest handler.
- Guests cannot rely on privileged CLI or STI instructions at ring 3, so they change interrupt masks by writing to the shared info page.
- Under heavy network load, a guest driver can mask event channels and poll shared requests instead of paying for frequent upcalls.
- The poll hypercall lets an idle guest sleep in the hypervisor until an event channel becomes pending, then check for events on return.
- Atomic synchronization primitives prevent races between sleeping and event delivery.
- Xen can expose an RDTSC counter, guest-only virtual time that advances while the guest runs, and wall-clock time synchronized by Domain 0.
- A guest application uses the unmodified Linux TCP/IP stack, which sends device requests through the Xen netfront driver.
- Netfront places requests in a shared producer-consumer ring and identifies guest pages through grant references.
- It batches requests and notifies the netback backend in Domain 0 through an event channel, reducing notification overhead.
- Netback claims grants and asks Xen to map guest pages into Domain 0; Xen also supports page copying, transfer, and page-flipping approaches.
- Page mapping can be expensive, so Xen implementations also used pre-shared memory regions and copied small packets when copying cost less than remapping.
- Domain 0 can bridge packets to other VMs, route them through Linux networking tools such as iptables or Open vSwitch, or send them through a physical NIC and DMA.
- Keeping physical device drivers in Domain 0 limits the damage from a driver bug; a compromised backend VM can disrupt networking without automatically compromising the hypervisor.
- Drivers are large and depend on hundreds of Linux kernel services, making it difficult to move existing vendor drivers into a small hypervisor.
- The approach preserves reuse of Linux drivers, including complex or partly proprietary drivers, while trading some performance for isolation and compatibility.
- A Xen grant table lets one domain authorize another domain to access a page, but the discretionary sharing model can permit unsafe sharing if domain identities are misused.
- Grant operations include mapping a page, transferring ownership, and copying data through the hypervisor; page flipping can exchange pages while preserving each domain’s page count.
- Xen rings place request and response entries plus producer and consumer indices in shared memory, and their layout can be tuned to reduce cache-line bouncing between cores.
- Xenstore is a Domain 0 process that provides guests with a configuration database and device-discovery interface instead of a PCI BAR-based device abstraction.
- Guests can watch configuration keys for changes, enabling device hot-plug notifications and updates to memory or event-channel assignments.
- Xenstore’s central role and relatively slow operations created security and performance concerns; PCI-based interfaces such as virtio became more common.
- Early Xen results reported near-native Linux performance for send and receive with large packets around a 1,500-byte MTU.
- Small packets reduce efficiency because they require more packet-processing work per unit of data; receive paths also pay for page handling and dispatch.
- The comparisons used older gigabit Ethernet hardware and historical baselines, so they do not directly describe modern 10, 40, or 100 Gbps systems.
- Grant mapping can require hypercalls and page-table updates, while a shared-memory design trades remapping for packet copies.
- Shared ring indices can bounce cache lines between cores, costing hundreds of cycles, with cross-socket communication adding further latency.
- SR-IOV lets a NIC expose virtual functions that can be assigned to VMs, enabling near-native device access and reducing reliance on the software frontend/backend path.
- DPDK illustrates the cost of very fast packet processing: saturating 100 Gbps with 64-byte packets can require three continuously busy CPU cores.
- The class begins planning a paravirtualized xv6 implementation for Homework 3.
- The design questions include how to boot and run guest kernel and user code, enforce isolation, handle privileged instructions, deliver interrupts, and manage page tables.
- The exercise is framed around choosing which hardware interfaces to emulate and which to replace with explicit hypercalls or shared-memory protocols.
- Changing CR3 to select a guest page-table root requires a controlled hypercall rather than executing a privileged instruction directly.
- Loading an interrupt descriptor table could be replaced by a hypercall that registers guest interrupt handlers.
- Simple state such as an interrupt-enable flag can live in shared memory, avoiding a hypercall for every CLI- or POPF-like operation.
- Paravirtualized console or disk interfaces can replace port I/O with explicit hypercalls.
- On x86-64, the design cannot rely on ring 1 for the guest kernel, so host and guest code must share the available privilege levels carefully.
- The proposed layout places guest user and guest kernel code in separate address regions while keeping the host kernel mapped at a higher address.
- Separate guest user and guest kernel page tables prevent user code from directly accessing guest-kernel mappings.
- A guest system call exits to the host kernel, which switches to the guest-kernel page table before entering the guest kernel, adding overhead but preserving isolation.
- A simple single-core guest can omit IOAPIC and LAPIC emulation, along with guest interprocessor interrupts.
- More advanced interrupt routing would require explicit hypercalls to configure virtual interrupt delivery.
- The simplification follows the earlier xv6 homework approach of omitting unnecessary hardware behavior rather than emulating it.
- The host tracks whether guest interrupts are enabled and consults the guest’s registered interrupt descriptor table when injecting an event.
- Interrupt delivery requires transferring control to the guest handler and arranging the guest execution state, not merely branching to a handler address.
- The design must preserve enough state for the guest to resume correctly after its interrupt handler returns.
- A hardware-style transition from guest user mode must save the guest RIP and flags on a guest kernel stack before entering the handler.
- The guest can register a kernel stack through a TSS-style mechanism or explicitly provide the stack pointer when switching processes.
- For interrupts arriving while guest kernel code is already running, the existing guest stack pointer can be used to save the required state.
- The class leaves additional HW3 design questions for the next meeting and asks students to submit their current activity work.
Transcript excerpt
0:37 Okay. So the plan today is to quickly finish uh discussion of the Zen per virtualization architecture and then uh start discussing uh how we can implement how can we can per virtualize X6 uh just to make sure that we understand how everything works. So what uh where we stopped last time, we uh I was explaining ballooning and uh like as a mechanism to essentially force the gas to release some memory to the host uh like in some like in some intelligent way meaning that a balloon which lives inside just allocates memory from the guest kernel and supposedly guest kernel
1:24 is designed to to react uh like memory pressure in some meaningful way, right? So it's like your guest policy, right? But uh what we haven't discussed yet is how like again Kazan delivers interrupts and uh what it does with device drivers or we we we did talk about device drivers earlier in the class. So essentially uh like everything is power virtualized, right? So uh and Gen essentially says okay guys you not not never like everything will be like will be simplified and architecture agnostic and essentially like we will deliver um instead of like providing a full
2:10 interrupt functionality and supporting like a full interrupt table we'll just be using we'll be just be using an abstraction of uh virtual ual event channel. And uh this virtual event channel is uh something what allows the guests to uh receive uh like uh like essentially like receive some notifications from device drivers, send interprocessor interrupts essentially allowing one one like because virtual machines will will run multiple CPUs, right? For performance reasons. uh and but also like uh like allow one domain to notify one virtual machine to
2:58 notify another virtual machine because
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.