Introduction to Nydus Image Service on In-kernel EROFS

### Overview In this presentation, Xiang Gao and Xing discuss integrating the Dragonfly Nydus image service with the Linux in-kernel EROFS (Enhanced Read-Only File System) over FS-Cache. Traditional OCI container image formats, which rely on tar.gz archives and layer-based stacking, incur major performance bottlenecks during cold start: pulling packages can account for roughly 76% of startup time, even though typically only 6.4% of that data is ever read. To address these inefficiencies, Nydus implements metadata/data separation and lazy pulling. The speakers explain how migrating Nydus from a user-space implementation (e.g., FUSE) to an in-kernel EROFS and FS-Cache architecture significantly cuts context-switching overhead and improves reliability in production environments.

### Key Topics - **EROFS File System Ecosystem:** Originally merged in Linux 4.19 and formally landed in 5.4, EROFS is a block-based, read-only file system featuring tail-packing, transparent LZ4/LZMA compression, and broad distribution support across Android, embedded systems, and server distros. - **Limitations of OCI Images:** OCI images suffer from coarse layer deduplication, non-seekable archives, missing runtime data verification per chunk, and overhead caused by overlayfs layer stacking. - **Nydus Architecture:** Nydus separates container image data into content-addressable chunk blobs and a unified metadata bootstrap layer, enabling chunk-level deduplication, runtime data integrity verification via chunk hashes, and on-demand decompression. - **EROFS over FS-Cache:** Replacing user-space format parsers and FUSE-based fetching with EROFS and kernel FS-Cache (introduced in Linux 5.19) allows local cache hits to be handled purely within kernel space. - **Daemon Failure Recovery and Reliability:** Solutions developed by ByteDance to handle user-space daemon (`nydusd`) crashes, restarts, or upgrades without causing container I/O errors or hangs.

### Technical Details - **In-Kernel Architecture:** Using an in-kernel container image format (referred to in the presentation as RAFS v6 based on EROFS) and FS-Cache decouples lazy pulling into kernel-side cache management and user-space network data fetching. When a container requests image data, FS-Cache serves cache hits directly in kernel space. On a cache miss, the kernel pauses the requesting process and notifies `nydusd` through `/dev/cachefiles` (`devfd`), which fetches the chunk remotely, populates the cache file, and wakes the sleeping thread. - **Optimizations:** Nydus supports asynchronous prefetching (downloading unrequested chunks in the background) and network I/O amplification (e.g., downloading 1 MB instead of a strictly requested 4 KB) to mitigate round-trip network delays while keeping subsequent reads in the kernel. - **Reliability and Daemonless Operations:** - *Local Requests & `devfd`:* To prevent FS-Cache from dropping into a "not present" error state if the user daemon exits, `devfd` is handed off to a supervisor process over a Unix domain socket. - *Anonymous FDs:* Rather than persisting many anonymous cache file descriptors in user space, kernel patches allow on-demand requests to trigger recreation and retransmission of anonymous FDs upon daemon restart. - *In-Flight Requests:* To prevent lost pending requests from surfacing as container I/O errors during unexpected daemon termination, requests are placed in an I/O wait state and re-queued via a restore command once `nydusd` recovers. - **Demo Benchmarks:** A container startup comparison demonstrated that a Nydus container started in 16 seconds versus 27 seconds for standard OCI, despite the workload reading a substantial amount of data immediately upon start.

### Takeaways Integrating Nydus with in-kernel EROFS and FS-Cache eliminates user-kernel context switching on cache hits and delivers notable cold-start latency reductions. With supervisor-managed descriptors and kernel recovery mechanisms for anonymous file descriptors and in-flight requests, container runtimes maintain high availability across daemon restarts. Future work includes in-kernel page cache sharing across deduplicated chunks, a fully daemonless in-kernel FS-Cache implementation, FS-DAX integration for memory sharing in Kata Containers, and lazy pulling support for standard OCI blobs.