Improving OOM debugging

### Overview In this session from the Linux kernel memory management (MM) room, speaker Kent Overstreet discusses the challenges of debugging out-of-memory (OOM) events and allocation failures in the Linux kernel (demonstrated on 5.17 with an unexpected order-5 allocation failure from 9p/virtfs). The presentation highlights how current OOM reporting tools like `show_mem` provide poor visibility into memory consumption and shrinker activity. Overstreet and attendees debate potential improvements to kernel reporting, rate limiting, pretty printing, and low-overhead tracking of memory allocations by call site.

### Key Topics - **OOM and `show_mem` Reporting Deficiencies**: Current OOM logs either dump overwhelming, unsorted output (such as pages of slab info) or omit critical context like shrinker states and unaccounted page allocations. - **Shrinker Visibility and Pretty Printers**: Incorporating shrinker information into reports and debugfs (referencing Roman's debugfs work) so subsystems can expose internal reclaimable/dirty state via callbacks. This ties into moving kernel pretty printers out of `vsnprintf.c` dumping grounds into typed helper functions. - **Report Formatting and Rate Limiting**: Discussions around rate-limiting OOM reports to prevent slow serial consoles from stalling machines, sorting top memory consumers, distinguishing kernel versus user space usage, and highlighting fragmentation versus outright exhaustion. - **Always-On Call-Site Allocation Tracking**: Proposals for a lightweight memory allocation profiling mechanism using an ELF section trick (similar to dynamic debug) to identify allocation call sites by file and line without the prohibitive overhead of stack-tracing tools like `page_owner` or tracing.

### Technical Details - **Pretty Printing Architecture**: Rather than relying on obscure `%p` extensions in `vsnprintf.c`, Matthew Wilcox and Kent propose a syntax (such as `%(`) where printf passes function pointers directly alongside arguments (e.g., calling `bdev_name`), enabling discoverability through tools like Cscope. - **Shrinker and Allocation Diagnostics**: Attendees note that shrinkers may not explain every failure; high-order allocation failures frequently stem from memory compaction failure or fragmentation rather than lack of reclaimable pages. Additionally, allocating pages directly via `alloc_pages` without explicit accounting to `vmstat` creates large unaccounted gaps between total, free, and known memory consumers. - **Dynamic Debug ELF Section Mechanism**: Traditional solutions like tracing or BPF maps encounter issues with user-space memory overhead, interrupt context concurrency, and recursing allocations during tracing. Kent proposes defining a static struct in an ELF section via macro wrapping calls like `kmalloc` and `alloc_pages`. The pointer to this struct would be passed down into allocators (leveraging existing metadata provisions similar to `page_owner`) to record allocation file and line numbers with minimal overhead, decrementing counters upon freeing. - **Downstream Script Compatibility**: Audience members note that enterprise environments (such as Google internal tooling) use automated log-parsing scripts that inspect OOM messages, though Kent argues kernel log messages should primarily serve human debugging rather than strict ABI stability.

### Takeaways Debugging OOM conditions remains painful even in minimal test setups like single VMs running `xfstests`. While existing tools like `page_owner` or tracing are valuable for reproducing issues, they impose too much memory and CPU overhead for production. A lightweight, always-on mechanism tracking allocations to file and line numbers—combined with concise, sorted OOM summary dumps and shrinker callbacks—would make memory consumption bugs visible and actionable across development and production environments.