BPF local testing (Manu Bretelle)

### Overview In this talk, Manu Bretelle covers practical techniques and personal workflows developed while working on BPF continuous integration (CI). Testing BPF programs and kernel changes locally often poses risks such as system crashes, and reproducing CI-specific test failures can be difficult when environments or hardware architectures differ. Bretelle presents methods to safely and rapidly iterate on BPF tests locally, pull CI artifacts to reproduce failures in identical environments, and set up cross-compilation toolchains to debug issues occurring on foreign architectures like s390x.

### Key Topics - **Fast Local Development and Iteration**: Accelerating kernel and self-test verification cycles without risking host system instability. - **CI Reproduction Environments**: Fetching remote GitHub CI test runs, kernels, and root filesystems to recreate exact CI failure scenarios locally. - **Cross-Compilation for Foreign Architectures**: Configuring multi-architecture tooling, cross-compilers, and emulation to diagnose target-specific failures such as endianness incompatibilities.

### Technical Details - **Local Testing Setup**: Testing locally relies on VM-based execution using Daniel's `vmtest` tool (version 0.2 or latest). The virtual machine filesystem requires QEMU Guest Agent (`qemu-guest-agent`), alongside standard QEMU virtualization. Crucial runtime assumptions for BPF tests include having the `sysfs` BPF filesystem mounted and bringing up the loopback interface, without which certain network-related tests fail. - **Interactive VM Debugging and Fast Loops**: By passing a dash (`-`) to `vmtest`, developers obtain an interactive shell inside the VM that shares the host filesystem. This setup allows running local diagnostic tools within the VM environment and enabled diagnosing a flaky test issue where unexpected ICMPv6 packets passed through. Furthermore, developers can edit and recompile self-tests on the host machine without rebuilding the entire kernel or exiting the running VM. - **Automated Bug Hunting with Git Bisect**: Bretelle highlights using `git bisect run` combined with `vmtest`. A bisect script builds the kernel and self-tests, runs the reproducing command via `vmtest`, and relies on `vmtest` returning the command's exit code (0 for good, 1 for bad, 125 for skip) to locate regressions automatically. - **Replicating GitHub CI Failures**: To mimic the CI environment, developers can retrieve CI run artifacts using the GitHub CLI (`gh run download`) targeted by Run ID to acquire the pre-built kernel (`bzImage`) and self-test binaries. For the filesystem, Bretelle pulls the exact Ubuntu container image used in CI into a root filesystem either via Docker (`docker create` and `docker export`) or using `docker2rootfs` (a custom utility written to extract Docker images without running Docker). The root filesystem and kernel are then booted using `vmtest -k <kernel> -r <rootfs>`. - **Cross-Compilation (e.g., s390x)**: Debugging foreign architecture issues on an x86 host requires adding target architecture repositories via `dpkg`, installing cross-compilers (`gcc`, `g++`), `libelf-dev`, `libssl-dev`, `pkg-config`, and `qemu-user-static` (leveraging `binfmt_misc`). A one-line patch is required in the self-test Makefile to avoid using the host's `bpftool` for `vmlinux.h` generation via BTF dump, which otherwise causes endianness mismatches between x86 and s390x. Target root filesystems are extracted from foreign architecture Docker base images, chrooted into, and populated with required test utilities.

### Takeaways - Utilizing `vmtest` with host filesystem sharing isolates the host from kernel panics while preserving local developer tooling and fast iteration. - Automated bisection can be fully integrated with VM-level error reporting. - Cross-architecture BPF testing requires careful handling of endianness during `vmlinux.h` generation and target rootfs preparation. Notes and workflows are published at `sandhose.github.io/bpfcitools`.