Skip to content

DEF CON 34 talks are here: 354 talks, summarized and searchable.

Browse DEF CON 34

An Engineer's Guide to Linux Kernel Upgrades

Ignat Korchagin

Supply ChainTalks52 minIntermediateOpen on YouTube ↗

Where

Talks
A New HOPE

Duration

52 min

Explains why Linux kernel updates are less risky than feared and how to schedule them safely in production.

Summary

Many production teams delay Linux kernel upgrades because they treat the kernel as the most dangerous component to change. The talk argues that this fear is misplaced and that delay carries its own costs. Using data from the 5.15 stable branch, it shows that weekly bug-fix releases often contain dozens to hundreds of commits, and that most include at least one CVE fix. Waiting also widens the gap between the running and target versions, so breakage risk grows faster than the delay itself.

It then covers common anti-patterns: routing kernel changes through generic change-approval processes, asking whether each CVE is exploitable instead of patching, and soaking kernels for fixed periods rather than using measurable criteria. Kernel version numbers are not semantic versioning, so the leftmost number says little about risk. Bug-fix releases carry only fixes, while major releases bring features on a roughly nine-to-ten-week cycle. Live patching covers only a narrow set of cases because internal kernel data structures can change.

The takeaways are to apply bug-fix releases promptly, choose a long-term branch if validating each major version is too costly, and write acceptance tests covering the kernel features each workload depends on. Decisions should rest on metrics and test results rather than elapsed time. Automating rollouts also removes the hesitation individual engineers feel, and reboot-driven rollouts are naturally gradual, giving time to catch regressions.

Related talks