DO-330 Qualification of Enhanced LLVM Structural Coverage Tool

### Overview Bingji Park and Sajin Kim, software engineers at Boeing Korea, present their work on qualifying an enhanced LLVM structural coverage toolchain under the DO-330 standard. Because Boeing decided to use Linux across various airborne avionics systems, the software must comply with DO-178C, a mandatory safety-critical standard governing avionics software development and verification across authorities such as the FAA (United States), EASA (European Union), and Transport Canada. Commercial structural coverage tools are costly, prompting the team to adapt and qualify the open-source tool `llvm-cov` to satisfy strict civil aviation verification objectives.

### Key Topics - **Avionics Standards (DO-178C & DO-330)**: DO-178C classifies software criticality from Level A (highest, failure could cause loss of life) down to Level E (no safety impact). DO-330 governs the qualification of tools used across the software life cycle. Tools are assigned Tool Qualification Levels (TQL-1 through TQL-5). Because `llvm-cov` is used for verification rather than source code generation, it falls under Criteria 3, resulting in a TQL-5 classification. - **Structural Coverage Requirements**: DO-178C mandates structural coverage for Levels A, B, and C: Level C requires statement coverage; Level B demands statement and decision coverage; Level A requires statement, decision, Modified Condition/Decision Coverage (MCDC), and object code coverage. - **LLVM Enhancements & Upstream Contributions**: Standard `llvm-cov` supported line and branch coverage. Boeing collaborated with the University of Illinois Urbana-Champaign (UIUC) to develop MCDC support, which was contributed upstream to LLVM and released starting in LLVM version 18.1.

### Technical Details & Toolchain Architecture The speakers address key gaps between native LLVM coverage primitives and DO-178C definitions: - **Statement Coverage via `clang-format`**: Native `llvm-cov` tracks line coverage rather than individual statement execution. If multiple statements are placed on a single line, line execution can mask unexecuted statements. The team integrates `clang-format` as a mandatory pre-processing step to enforce exactly one statement per source line. - **Decision Coverage via Branch Coverage**: Native `llvm-cov` provides branch coverage (evaluating branches at each condition) rather than decision coverage (evaluating the overall Boolean decision outcome). Because branch coverage is mathematically more rigorous, achieving 100% branch coverage guarantees 100% decision coverage. - **Conceptual Tool Pipeline**: The entire qualified verification toolchain is treated as a single conceptual tool under DO-330: source code is formatted with `clang-format`, compiled and instrumented via `clang`, executed to generate raw profile data, indexed using `llvm-profdata`, and finally processed by `llvm-cov` to produce coverage reports. - **DO-330 Qualification Artifacts**: TQL-5 compliance requires tool operational requirements (TOR), verification/validation cases, procedures, and test results. Boeing is also assembling optional artifacts for cross-project reuse, including the Tool Qualification Plan (TQP) and Tool Accomplishment Summary (TAS).

### Takeaways & Future Work Tool qualification cannot be submitted in isolation; it must be submitted to certification authorities (such as the FAA) alongside the actual airborne software product utilizing it. If the toolchain version or deployment environment changes, re-qualification is required. Looking ahead, the team and UIUC are developing object code coverage to verify compiler-injected instructions for Level A software, alongside long-term efforts to measure structural coverage of the Linux kernel.