The SBOM Challenge

### Overview At the S4 conference, researchers from the Idaho National Laboratory (INL)—including co-speaker Hannah Pearson Kleinheiter and her co-presenter Ginger—presented the findings of the first Software Bill of Materials (SBOM) Challenge. Hosted in collaboration with Dale, the competition evaluated five commercial and open-source tooling providers across three core objectives: extracting component/file SBOMs from artifacts, enriching components with vulnerability data (CVEs/non-CVEs), and ingesting Vulnerability Exploitability eXchange (VEX) documents. Rather than revealing a single "silver bullet," the challenge demonstrated that tooling performance strongly reflects differing core metaphors, architectural priorities, and intended use cases across the participating platforms.

### Key Topics - **Participating Tool Vendors:** Five entities took part in the evaluation: aDolus (FACT platform - Framework for Analysis and Coordinated Trust), Cybeats, Finite State, NetRise, and Security Risk Advisors (SRA, utilizing their own open-source FACT tool - Firmware Analysis and Comparison Tool). - **Test Artifact Diversity:** INL provided three distinct artifacts representing real-world ICS and operational environments: Hitachi Energy's Transformer Protection Unit (TPU 2000, a minimal bare-metal 68k firmware binary), OSIsoft ProcessBook (a Windows-based data viewing application), and OpenWrt (an ARM-based open-source router firmware with compiled libraries requested by participants). - **Naming Collisions and Ambiguity:** The speakers highlighted persistent industry-wide naming challenges for software components, noting that even within the challenge itself, two different participants (aDolus and SRA) shared the identical platform acronym "FACT." - **Standardization & Formats:** Tools primarily interacted with CycloneDX and SPDX formats, alongside automated vulnerability sharing via VEX documents.

### Technical Details - **Phase 1: Component Extraction:** Tools faced varying success based on architecture. Bare-metal firmware (TPU 2000) proved unhandled by all tools; zero components were extracted from it. For OSIsoft ProcessBook, NetRise rapidly built Windows compatibility during the competition to extract components. OpenWrt yielded significantly higher package and file counts across participants. Cybeats did not perform raw binary component extraction, focusing instead on SBOM ingestion, structure validation, and distribution. - **Phase 2: Vulnerability Enrichment:** Participants correlated extracted components against multiple vulnerability data sources (such as NVD and international databases). High variance occurred in reported numbers; for instance, Finite State identified 85 vulnerabilities in ProcessBook tied largely to a single library, and also reported non-CVE vulnerabilities. INL stressed that a vulnerability existing in an upstream library does not guarantee exploitability within the compiled product. - **Phase 3: VEX Ingestion:** INL supplied synthetic, fictitious VEX documents to test status updates without misrepresenting vendor security. Finite State and SRA successfully ingested the VEX files to adjust risk status. aDolus declined ingestion under their policy that only verified OEMs/vendors hold the authority to issue VEX documents. NetRise did not support VEX due to a lack of market customer requests, while Cybeats demonstrated capabilities to author and structure VEX documentation. - **Ad Hoc Findings:** Beyond SBOM generation, NetRise detected a hard-coded password in one artifact, which was verified with the originating vendor.

### Takeaways INL concluded that prospective buyers must define their organizational "quest" and specific use case before selecting an SBOM tool, evaluating hands-on user interfaces rather than relying solely on raw metric counts. Moving forward, INL proposed longer evaluation periods (weeks to months, modeled after CISA plugfests), examining full vendor catalogs, and establishing an energy sector SBOM exchange program under legal/liability protections (such as NDAs or DOE frameworks) to help asset owners and vendors mature SBOM consumption.