Scratch containers and the supply chain trouble

### Overview In this presentation, Naveen, a senior security engineer at Freshworks, explores the security implications, operational trade-offs, and supply chain challenges of building container images using the Docker `scratch` base image. While `scratch` containers significantly reduce image size and attack surface by removing extraneous OS packages and shells, they introduce serious blind spots for vulnerability scanners, creating a false sense of security regarding software supply chain health.

### Key Topics - **Base Image Dilemmas:** Standard parent images (such as Ubuntu or Debian) are frequently oversized, outdated, and burdened with unpatched CVEs. While reducing images to only necessary dependencies limits exposure, standard multi-stage builds using `scratch` represent an extreme approach where no parent image layer is added. - **Compiled vs. Interpreted Languages in Scratch:** Statically compiled binaries (such as Go) run naturally inside `scratch` containers once stripped of external C dependencies (`CGO_ENABLED=0`). In contrast, interpreted languages like Python require workarounds using packaging tools like PyInstaller and StaticX to bundle dependencies into a statically linked binary. - **Scanner Blind Spots & Inconsistencies:** Traditional container scanners (e.g., Trivy) typically rely on OS package managers (`apt`, `dpkg`, `apk`) or language package manifests (`go.sum`, `pip`). In `scratch` containers lacking these package managers and metadata structures, scanners frequently report "zero vulnerabilities" despite the presence of known vulnerable application and system dependencies. - **Distroless as an Alternative:** When `scratch` presents excessive build complexity or incompatibility, distroless base images (such as Chainctl/Chainguard Wolfi images or Bazel-built minimal images) offer a middle ground without package managers or shells while retaining required system dependencies.

### Technical Details - **Go Application Test:** Naveen demonstrated a Go web server using a vulnerable Gin 1.4 dependency. Built on Debian Bullseye, Trivy detected 675 vulnerabilities (667 from Debian, 8 from the Go binary). Testing with Alpine showed zero base OS vulnerabilities and only the 8 binary vulnerabilities. Building into `scratch` also surfaced the 8 application vulnerabilities because Go language parsers can extract embedded module metadata from binaries. - **Python Application Test:** Naveen packaged a Python application containing a vulnerable `requests` package (version 2.3) using PyInstaller and StaticX. On Debian Slim, Trivy detected 81 vulnerabilities; on Alpine, it reported 0. When placed in `scratch`, Trivy reported 0 vulnerabilities because it failed to recognize the bundled binary or package format, completely missing the known CVE. - **Scanner Detection Mechanics:** Vulnerability tools detect packages via package managers, layer hashes from manifests, SBOM attestations, or language-specific decompilers/debuggers. Static linking of shared objects (`.so`) into standalone binaries hides system-level dependencies from tools that rely on OS package databases. - **Exploitability vs. Compliance:** Even though missing runtime environments (like shells or utility binaries) can make remote code execution or injection exploits fail or simply terminate the container, underlying business logic and vulnerable dependencies remain unmonitored if scanners report false negatives.

### Takeaways A report of zero vulnerabilities from a container scanner does not guarantee an application is secure or compliant. Organizations using `scratch` containers must perform due diligence, incorporate build-time SBOM generation before binary packaging, and be cautious of hidden vulnerabilities within statically linked shared objects.