Securing Container Image Supply Chains with tools such as Goss and OpenSCAP

## Overview Chris Howarth from Citi presents methods for securing container image supply chains, focusing on two open-source tools: Goss and OpenSCAP. The talk emphasizes that as over 75% of organizations run containerized applications, consuming external third-party container images creates significant security risks requiring validation at build time.

## Container Image Structure & Supply Chain Risks Container images consist of three layers: a stripped-down OS base layer, middleware (e.g., Java, Nginx), and the application itself. Most organizations consume externally provided base and middleware layers. Recent software supply chain attacks—notably CodeCov (where a Docker image processor was infiltrated and modified to exfiltrate credentials) and SolarWinds/Sunburst—demonstrate the critical need for image verification before deployment.

## Goss: File and Policy Validation Goss is a lightweight, single-binary testing tool that is OS-agnostic. The demo shows two primary check types: - **File integrity checks**: Creates YAML fingerprints of critical files (e.g., /etc/passwd) containing owner, group, and permission metadata. Running `goss validate` confirms integrity; any tampering (e.g., ownership change to 'games' instead of 'root') triggers failures. - **Command checks**: Uses simple grep-based tests to validate application configuration. For example, checking that an application's cipher is set to AES-256 in config files. Goss captures exit status and output to detect unauthorized modifications.

## OpenSCAP: Compliance & Policy Benchmarks OpenSCAP provides pre-canned compliance policies based on industry standards such as CIS Benchmarks and DoD STIG profiles. It generates human-readable HTML reports showing pass/fail status for each test along with remediation guidance. Installation is more complex than Goss (includes dependencies), and custom policies can be created using the Workbench tool, though this is more involved than Goss's template-based approach.

## Findings from Policy Checks Common security issues detected include: - World-writable files and directories - Shell access configured for container accounts (e.g., `/bin/bash` instead of `/sbin/nologin`) - Careless PATH settings that include application directories, allowing malicious scripts to override system binaries like `rpm`

## CVE & Vulnerability Scanning Vulnerability counts in images grow rapidly over time (e.g., jumping from 10 to 80 critical/high vulnerabilities within months). Rapid patching and re-deployment of images built on the latest base layers is essential to minimize exposure window.

## End-to-End Secure Pipeline The speaker describes a fully automated pipeline: 1. Pull base image from external repository 2. Verify image integrity 3. Add organizational customizations (certificates, repos, hardening) 4. Validate Dockerfile using conftest 5. Run Goss and OpenSCAP tests 6. Perform CVE scanning 7. Push validated image to production 8. Repave dependent layers with updated base image

Using multi-stage Docker builds, final images exclude test tooling, reducing attack surface. Tools like Podman and Kaniko are preferred over Docker as they execute in user space rather than as root.

## Key Benefits & Successes - **Reproducibility**: Fully automated builds ensure consistent, security-focused image construction - **Speed**: Organizations achieved same-day deployment of images after third-party releases, drastically reducing CVE exposure windows - **Robustness**: Pipeline components include automated tests validating the tests themselves - **Trust**: Comprehensive validation and scanning provide high confidence in image security posture

## Q&A Insights When asked about malicious base images potentially compromising Goss/OpenSCAP results, the speaker noted that Dockerfile validation via conftest before running the pipeline helps prevent tampering. Podman's user-space execution limits damage from a compromised build environment to user level rather than root.