cs6465 Advanced OS - Lecture 07 - Physical and virtual devices
Watch on YouTube →
Overview
Anton Burtsev explains how Linux discovers and communicates with physical PCI devices, using the 1 Gbps Intel E1000 network interface to show MMIO, BAR allocation, DMA rings, doorbell registers, and interrupts. He then compares full E1000 emulation, which requires costly VM exits, with Xen-style split drivers and Virtio queues, which share memory and support asynchronous, potentially out-of-order I/O with fewer exits.
Key takeaways
- PCI device discovery and device communication are separate stages: firmware or Linux enumerates buses and configuration headers first, then sizes and assigns BAR regions before a matching Linux driver can probe the device.
- The E1000 uses DMA descriptor rings for packet data and MMIO head and tail registers for coordination; advancing the tail acts as a doorbell, while completion bits and head updates let the driver reclaim buffers.
- Full E1000 emulation requires intercepting PCI configuration, BAR accesses, DMA, and interrupts, so frequent MMIO operations can trigger costly VM exits.
- Xen split drivers and Virtio reduce critical-path exits by exchanging requests through shared-memory queues, with grant-table mappings and batched hypercalls used where page sharing must be established.
- Virtio separates available requests from used completions and chains descriptors for scatter-gather I/O, enabling out-of-order completion that is useful for both network and block devices.
Chapters
- The homework deadline is moved until after spring break so students have roughly two weeks to complete it.
- Anton Burtsev says most CAD machines support Intel VMX, allowing the assignment to run with KVM; cloud machines are also an option.
- Mac users can use the provided Bochs-based emulator, though it does not support connecting GDB.
- Burtsev shifts from prior discussion of paravirtualized drivers to the fundamentals of operating-system communication with physical devices.
- The Intel E1000 is a simple 1 Gbps network interface used as the example, despite modern interfaces reaching 100 or 200 Gbps.
- Virtio applies ideas from Xen split drivers in a more general form for modern virtualization.
- With memory-mapped I/O, the CPU issues ordinary loads and stores to special address ranges that the host bridge routes to PCI devices instead of DRAM.
- Port I/O uses x86 IN and OUT instructions and remains a legacy alternative to MMIO.
- A device can notify the CPU through an interrupt routed via the APIC and IDT, or move data directly between device and memory using DMA.
- Firmware or the operating system scans PCI configuration space to identify devices; Linux may rescan rather than rely entirely on BIOS results.
- Configuration headers expose fields such as vendor ID and device ID, which identify manufacturers such as Intel and specific hardware models.
- The scan establishes device locations first; BAR regions must be configured before drivers can communicate with devices.
- PCI topology consists of buses, devices, and functions, with up to 256 bus numbers and 32 device positions per bus in the described scheme.
- Enumeration begins at bus 0, checks its device slots, and assigns bus numbers while following bridges to downstream buses.
- A depth-first traversal finds devices such as GPUs, NVMe SSDs, network cards, and the E1000 while recording their bus locations.
- A device’s Base Address Registers, or BARs, describe the address regions used to access its registers; devices may expose multiple BARs.
- Firmware or the operating system writes all ones to a BAR and reads back a mask to calculate the required region size.
- In the example, the E1000 BAR requires 128 KB; the system assigns an address, often within a reserved region below 4 GB, and writes it back to the device.
- A Linux PCI driver registers supported vendor and device IDs when its module initializes or when the kernel boots.
- When scanning finds a matching device, Linux invokes the driver’s probe function to begin initializing and managing the hardware.
- The `lspci` command displays PCI devices and their bus topology, including network interfaces and storage devices.
- The E1000 exposes device registers through a BAR, with offsets selecting registers such as ring base, length, head, and tail.
- The transmit and receive rings reside in ordinary system DRAM, and the driver gives the device their physical addresses through BAR registers.
- Each descriptor identifies a packet buffer and length and includes status bits, including a completion indicator that the device updates.
- For transmission, the driver fills descriptors with packet buffers and advances the tail register; that MMIO write acts as a doorbell telling the E1000 to process the new entries.
- The E1000 DMA-reads descriptors and packet data, then transmits the packet; it later updates completion status and the head position.
- For reception, the driver places available buffers in the receive ring and advances its tail so the device can DMA incoming packets into them.
- Interrupts can announce received or completed work, while polling modes such as DPDK repeatedly check ring state instead.
- The driver can reclaim completed E1000 transmit descriptors and packet buffers after the device advances its head and marks work done.
- Network transmit completions are described as in order, allowing earlier ring entries to be freed together.
- Disk devices may complete requests out of order; NVMe can report request IDs through a completion ring so one slow request does not block ring cleanup.
- A Linux `send` call passes data through the socket and TCP layers, which create a kernel `sk_buff` rather than keeping the application’s page pinned until completion.
- Copying user data into an `sk_buff` avoids the costs of pinning user pages for the duration of device I/O.
- The E1000 can represent packet fragments with separate descriptors, using flags to mark the beginning and end so hardware can assemble and transmit the packet.
- A virtual machine that runs an unmodified E1000 driver needs a virtual PCI bus, configuration-space scanning, and BAR sizing and assignment.
- The virtual machine monitor must emulate E1000 BAR accesses, DMA behavior, packet transmission and reception, and interrupts.
- Writes to a doorbell or reads of a device register can cause VM exits to QEMU or another monitor, adding substantial latency on the I/O critical path.
- Xen split drivers divide I/O between a frontend in the guest VM and a backend that handles the device, communicating through shared-memory rings.
- Shared ring entries and pointers let the guest submit work without exiting on every register write; the backend can map guest pages using grant tables.
- Mapping operations can be batched into fewer hypercalls, and interrupts, called events in Xen, are needed mainly when polling is insufficient.
- The design keeps PCI-like device discovery but avoids frequent exits during packet submission and completion.
- Virtio presents a device through PCI identifiers so the standard Linux PCI layer can discover it and bind a Virtio PCI transport driver.
- The transport connects to Virtio core code, which then initializes a device-specific driver such as Virtio Net.
- QEMU supplies the virtual device, while Virtio provides a common interface for network and block devices.
- Virtio queues use a descriptor table plus separate available and used rings to track submissions and completions.
- Available-ring entries refer to descriptor-table indices; descriptors can chain together to represent scattered packet fragments or other scatter-gather I/O.
- The used ring reports completed request IDs, allowing requests to complete out of order and letting the driver maintain a free list of reusable descriptors.
- Guest and backend can inspect shared Virtio ring memory directly instead of reading device registers over PCI for each completion.
- The driver can negotiate when to send virtual interrupts, and active polling can avoid interrupts when the backend is already checking the queue.
- Virtio supports both network and block I/O, including chained fragments and out-of-order completion; Burtsev identifies a Virtio device driver as a likely future homework assignment.
Transcript excerpt
0:32 Okay, let me share my screen. Uh, I have a question though. So, I did assign the homework, right? So, but I I don't think I realized that the next week is a spring break. Is it Am I right? So is it really the next week uh we don't have any classes and everyone is on break? >> Yes. >> Uh then probably we should move it right. So we should move it to the week after so people can uh can actually work on it. Uh and uh like I can I can do the office hours, right? So something like a Friday after the spring break. Will it work? It's just that I wanted to assign the
1:19 next one as well. Or if it's like super easy, then there is obviously no reason to move. But wait, I assigned it when I assigned it uh on over the weekend on last Friday or something. I probably need at least two weeks, right? Yeah, let's move it after the spring break. Uh after the fall break. Um, makes sense. And uh, are there any are there any like immediate questions about the homework? Right now, >> yes, I do have a question. Um, let's say if I want to use a cloud machine, does that is it going to work on that or not? >> It will, but you don't need to honestly. So like I did not check all the CAD
2:07 machines, right? But most of the CAD machines have uh hardware supported virtualization, right? which means that uh the instructions should work on Kate. I can specifically post the ones which I tried. >> But what what's your reason why do you think you need an Angulab machine? >> Uh not specifically. I mean I just kind
This is the start of the transcript. Save this video free for the whole transcript, searchable and time-coded.
In this course
Related notes
Summary, takeaways, and chapters were generated by AI from the video's transcript and may contain errors. The video belongs to its creator, Anton Burtsev.