Trusted-SBOM: On the Critical Importance of Verifying SBOMs

### Overview In this talk, Holly from the Center for Software Excellence at Huawei Canada, alongside co-speaker Professor Ahmed Hassan from Queen's University, explores the critical necessity of verifying Software Bills of Materials (SBOMs). While SBOM adoption is expanding—backed by initiatives such as OpenSSF's $3.2 million "SBOM Everywhere" project and defined by organizations like the Linux Foundation, NTIA, and CISA—the authors highlight that existing SBOM generation techniques (manifest-based, build tracing, semi-automated, or tools like Google Open Source Insights) frequently suffer from quality issues. Low-quality SBOMs lead directly to inaccurate vulnerability tracking via missed alarms (undeclared dependencies that are used) and false alarms (declared dependencies that are never used), while also compromising license management.

### Key Topics - **SBOM Quality Metrics**: Defining and formalizing "correctness" and "completeness" to evaluate SBOM accuracy against software binaries. - **Inaccurate Vulnerability Tracing**: Examining missed alarms caused by undeclared components versus false alarms caused by unused declared components. - **Real-World SBOM Verification**: Evaluating publicly available Java applications to determine empirical levels of SBOM correctness and completeness. - **Code Provenance Infrastructure**: Using large-scale provenance databases like World of Code and Software Heritage to identify unknown and undeclared files.

### Technical Details To evaluate SBOM quality, the speakers formalize two core metrics: - **SBOM Correctness**: The proportion of packages declared in an SBOM that actually exist in the software binary (number of verified packages divided by total declared packages). A package is considered verified if at least one of its files is present in the binary. - **SBOM Completeness**: The proportion of files in the software binary whose provenance can be accounted for (number of files traced to known dependencies or the software itself, divided by the total number of files in the binary).

Empirical examples illustrate practical risks: - **High-Cube (v1.0.5)**: Evaluated using Google Open Source Insights (OSI), which declared 11 dependencies. The verified correctness was only 0.27 (3 of 11 packages verified). For example, 31 of 33 files for `commons-logging` (v1.2) were matched, whereas no files or signatures for `log4j` (v1.2.17) could be found in the binary, resulting in four false-alarm vulnerability advisories. - **JUnit Platform Console (v1.8.2)**: Demonstrated a completeness score of 0.97. Out of 1,553 total files, 285 were verified across 6 dependencies, 1,268 were verified as internal software files, and 48 files remained unknown. Of those 48, 45 were Java source files under `org.hamcrest`, originating from undeclared dependency `hamcrest-core`. Such unlisted files create untraceable vulnerabilities and open vectors for supply chain attacks.

In a broader study of 100 public Java applications across three sources, the median SBOM correctness was just 0.31 (dropping to 0.07 when restricted to Java class files). The median completeness was 0.65.

### Takeaways The verification process for completeness currently relies on heuristics (e.g., matching application keywords in file names) and manual expert analysis, which limits scalability. The speakers recommend adopting large-scale code provenance databases such as World of Code and Software Heritage to resolve untraced files. Furthermore, the community needs to move beyond mere SBOM generators by developing verification tools, establishing standardized quality metrics, and creating approval organizations to certify trusted SBOMs similar to organic farming standards.