Monsters, Ghosts, and Bugs: How to Choose a Kernel

### Overview In "Monsters, Ghosts, and Bugs: How to Choose a Kernel," a Fedora kernel maintainer and Red Hat engineer explores how upstream Linux kernels and downstream distribution kernels are maintained, released, and secured. The talk prompts users and system administrators to think critically about why they run specific kernels, balancing trade-offs between cutting-edge features, long-term stability, and security maintenance workloads.

### Key Topics - **Upstream Release Cycle**: Upstream kernel development follows a predictable cycle managed by Linus Torvalds. It starts with a two-week merge window where new features land, followed by roughly seven to eight weekly release candidates (RCs) focused solely on bug fixes. During the merge window, maintainers like Greg Kroah-Hartman typically avoid general patch reviews due to extreme volume. - **Stable and LTS Trees**: Once a kernel version releases, it receives minor stable updates (e.g., 5.3.1, 5.3.2) containing self-contained, reviewed bug fixes (generally under 100 lines tagged with `CC stable`). Long-Term Stable (LTS) kernels are maintained for extended periods (typically one version selected annually by Greg Kroah-Hartman, such as 5.4, or others like 3.16 maintained by Ben Hutchings). - **Distribution Approaches**: Distributions select kernel strategies aligned with their philosophies. Arch and Fedora rapidly rebase to the latest mainline kernels, delivering modern features like eBPF enhancements, new container primitives, and updated hardware drivers, but carrying the risk of regressions (e.g., an ext4 filesystem corruption caused by a block layer regression). Ubuntu mixes LTS and rolling kernels, while enterprise distributions adopt curated, long-lived bases. - **"Franken-Kernels" vs. Well-Curated Kernels**: Enterprise distributions like Red Hat Enterprise Linux (RHEL) backport select features (such as eBPF) and specific bug fixes into older kernel versions rather than taking every upstream stable release. While critics label them "Franken-kernels" and warn that missing upstream commits could leave latent security bugs, the speaker describes them as well-curated kernels designed for strict binary compatibility and predictable stability, maintained by large teams of experienced engineers. - **Embedded and Android Evolution**: Historically, embedded systems and early Android devices shipped heavily out-of-tree, outdated kernels due to product shipping deadlines conflicting with upstream merge schedules. Over time, Android transitioned to using LTS branches as base trees, testing them heavily, and contributing testing results and code upstream.

### Technical Details & Security Considerations Maintaining custom kernels or out-of-tree patch sets carries heavy operational overhead, especially regarding security vulnerabilities like zero-days or hardware speculative execution flaws (Spectre and Meltdown): - Merging and backporting patches regularly causes merge conflicts as internal data structures and kernel APIs evolve. - Resolving merge conflicts without deep familiarity can introduce subtle errors; the speaker noted a past personal experience with Android kernels where an incorrect merge introduced a silent memory accounting bug displaying negative page counts. - When severe vulnerabilities occur, operators of custom trees must manually locate, backport, and verify security fixes, risking incomplete mitigation or regressions.

### Takeaways - For production systems, avoid maintaining standalone custom kernels unless backed by dedicated engineering teams with the capacity to review and backport fixes continuously. - If a custom kernel is necessary, base it strictly on an upstream LTS tree and regularly pull stable updates. - Choose a distribution kernel that matches your specific operational requirements: mainline rebasing for cutting-edge hardware and tracing features, LTS for low-churn deployments, or enterprise curated kernels for binary compatibility guarantees.