Fedora Automated Testing State of the Art: Not Quite CI for Distributions, but...

### Overview In this talk, Adam Williamson, a Senior Principal Software Quality Engineer and Fedora QA team lead at Red Hat, outlines the state and evolution of automated testing for the Fedora Linux distribution. He explains how Fedora shifted from a purely manual, VM-driven testing workflow in 2009 to a modern automated testing architecture. While full Continuous Integration (CI) across an entire operating system distribution remains impractical due to complex inputs, package interdependencies, and lengthy compose build times (taking around six hours), Fedora has implemented automated validation and gating mechanisms across different stages of the release pipeline.

### Key Topics - **Build and Distribution Pipeline**: Software packaging in Fedora moves from dist-git repositories to builds in Koji, bundled package updates in Bodhi, and full repository/image composes generated by Pungi. - **OpenQA for Functional Testing**: Adopted originally from openSUSE, openQA is leveraged for functional end-to-end testing. It automates virtual machine interactions over VNC and serial console, mimicking a user interacting with the GUI, installer, and desktop environments. - **Fedora CI for Package-Level Testing**: Built with Zuul and using the Testing Management Tool (TMT), Fedora CI allows packagers to define self-service test suites within dist-git repositories, running sanity checks and pull request tests. - **Update Gating**: Moving away from advisory-only tests or attempting to gate full composes, Fedora enforces automated gating at the update level in Bodhi for "critical path" packages. This prevents breaking changes from entering composes in the first place. - **Infrastructure and Orchestration**: Testing workflows rely on Fedora Messaging (AMQP bus), ResultsDB for test status storage, WaiverDB for manual overrides, and Greenwave for automated gating decisions.

### Technical Details - **OpenQA Capabilities and Mechanics**: OpenQA operates across x86_64, aarch64, and ppc64le hardware instances, supporting multi-machine test topologies (e.g., FreeIPA server, replica, and client configurations). Visual checks use "needles" (targeted sub-screen image regions) with configurable fuzz/matching thresholds (typically defaulting around 94%) to accommodate minor graphical rendering shifts. - **Validation Test Suite**: OpenQA automates roughly 75% of Fedora's release validation test suite, covering storage configurations (Btrfs, ext4, disk resizing, dual booting) and GNOME core desktop behaviors. - **Bodhi Update Testing**: For critical path updates, openQA replicates release engineering steps to generate miniature Live, netinstall, and Silverblue images, verifying installation integrity before an update can transition to stable. - **Results Integration**: Automated results surface via dist-git pull requests, Bodhi web interfaces, and legacy automated wiki tables, where remaining manual test results are merged with automated test outputs.

### Takeaways - Distro-level CI differs fundamentally from single-repository application CI because of the scale, build duration, and varied non-package inputs (such as Kickstart files, comps package groups, and compose tool configurations). - Gating package updates before they land in compose pipelines is significantly more effective and sustainable than attempting to gate final composes, preventing failure cascades across subsequent testing cycles. - Resource constraints and reliability requirements strictly limit gating scope; automated tests must be highly reliable to maintain packager trust without causing blocking false positives. Physical hardware tests and specific end-user validation tasks (around 25% of the test suite) intentionally remain manual.