QEMU Live Update

### Overview Steve Sistare presents QEMU Live Update, a mechanism developed at Oracle alongside Anthony Yznaga and Marc Conda to upgrade QEMU binaries in place while keeping virtual machine guests alive. Designed to deliver critical bug fixes, security mitigations, and new features without guest reboots, the technique avoids the heavy resource penalties of live migration—such as network/memory overhead, reduced fleet utilization, and long or indeterminate migration convergence times. Guest downtime during live update is minimal, causing only a brief pause of approximately 100 milliseconds without dropping external client connections.

### Key Topics - **Live Update Architecture:** Executing a new QEMU binary from the running QEMU process while carrying forward execution context and preserving RAM mappings in place. - **File Descriptor & Device Preservation:** Keeping external descriptors (consoles, VNC, monitor, vhost) and VFIO device descriptors open across `exec` while closing and recreating the KVM instance. - **Checkpoint and Restart (CPR):** Managing state serialization via the new `cprsave` and `cprload` monitor commands across QMP and HMP interfaces. - **Legacy Binary Updating via Injection:** Applying live update to older QEMU releases that lack native CPR commands by injecting a dynamic library (`vmsave.so`) into memory. - **Host Kernel Updates via Kexec (PKRAM):** Extending the CPR framework to reboot the underlying host kernel while preserving guest memory across `kexec`.

### Technical Details - **Memory Preservation:** The mechanism uses a proposed `madvise(MADV_DOEXEC)` extension (~300 lines of kernel code) that marks VMAs with a keep flag, copying anonymous memory (`MAP_ANON`) mappings across `exec` similarly to `fork`'s VM duplication. Executed binaries must explicitly authorize incoming mappings via an ELF note. Address and length information for RAM blocks (e.g., `pc.ram`) are passed via environment variables and reattached to the new KVM instance using `KVM_SET_USER_MEMORY_REGION` (which was patched in Linux 5.8 to eliminate a linear scaling bottleneck with page counts). - **VFIO and SR-IOV:** VFIO descriptors (container, IOMMU group, device FDs, event FDs) are carried over across `exec` without triggering re-initialization ioctls. MSI-X tables and pending bits are stored in a small (<1 MB) VM state file. Interrupt delivery continues seamlessly whether delivered to QEMU, the KVM IRQ chip, or posted directly to the guest. - **Update Process Workflow:** `cprsave` suspends guest VCPUs, serializes device state to a file, applies `madvise`, clears `O_CLOEXEC` on descriptors, destroys the old KVM instance, and calls `exec` (or a wrapper `/usr/bin/qemu-exec`). Once the new binary enters the pre-launch state, management layers can attach additional devices before `cprload` restores device states and resumes VCPUs. - **Legacy Injection:** For legacy QEMU versions, a small patch is written directly into `/proc/<pid>/mem` over a QEMU monitor function. It invokes `dlopen` on `vmsave.so` (matched to the target's GCC build ID) and resolves symbols via symbol table lookup, executing CPR functionality dynamically. - **Host Kernel Updates:** With the `reboot` mode in CPR, the host executes a `kexec` reboot. Guest memory is preserved using Anthony Yznaga's PKRAM (Preserved Kernel RAM) patches, which link and pass page frame numbers (PFNs) across reboot and restore shared memory segments in under a second for hundreds of gigabytes. SR-IOV across reboot is supported via guest agent suspend-to-RAM (`qemu-ga`).

### Takeaways Live update provides a lightweight, sub-second alternative to live migration for updating hypervisors and host kernels without severing network connections or disrupting guest operations. While upstream reviewers have hesitated on `madvise(MADV_DOEXEC)`, alternate approaches using `shmat` or `memfd_create` along with new VFIO ioctls are under consideration, alongside collaboration with Intel's VMM fast restart project for SR-IOV support across reboots without requiring guest agents.