Detecting the Undetectable: Threat Hunting in Appliance Environments

### Overview In this talk, Shahar Dorfman and Sagit Sadik, threat researchers from the Wiz research team, present a methodology for threat hunting in virtual appliance environments. Virtual appliances—such as solutions from Ivanti, Aviatrix, and Fortinet—are preconfigured, dedicated software packages where the vendor chooses the OS, application, and dependencies. While simple to deploy, they function as black boxes with immutable file systems, restricted root or SSH access, minimal logging, and strict restrictions preventing administrators from installing conventional telemetry or antivirus agents. Furthermore, appliances frequently run exposed to the internet with high privileges (for example, 65% of monitored Aviatrix instances in AWS held high IAM permissions), making them prime targets: 23% of all known exploited CVEs in enterprise environments were found in appliances.

### Key Topics - **The Visibility Challenge:** The inability of defenders to run endpoint agents, access files directly, or collect rich host telemetry on appliances. - **Snapshot Forensics:** Leveraging cloud volume snapshot APIs to image disks out-of-band and analyze file systems without relying on appliance OS access. - **File Prevalence Analysis:** Exploiting appliance uniformity across deployments to build path and hash baselines, isolating anomalous "long tail" files present on 5% or fewer machines. - **Real-World Case Studies:** Investigating recent vulnerabilities and uncovering web shells, Sliver C2 implants, and state-sponsored activity across PAN-OS and Ivanti EPMM devices.

### Technical Details To bypass appliance visibility restrictions, the researchers utilize cloud snapshot APIs to attach appliance disk snapshots to dedicated analysis systems as forensic images. Because appliance file systems are largely identical across deployments, file paths and SHA-1 hashes are aggregated across multiple customer environments to build prevalence histograms: - **100% Prevalence:** Stock legitimate vendor files. - **90–100% Prevalence:** Standard files varying due to OS or software versions. - **10–70% Prevalence:** Common optional add-ons or plugins. - **<= 5% Prevalence ("The Long Tail"):** High-confidence anomaly zones where malware typically resides.

In the first case study on Palo Alto Networks' PAN-OS, attackers chained two vulnerabilities discovered in late 2024 to achieve unauthenticated remote code execution. By filtering for low-prevalence files with suspicious extensions (such as PHP) or common files with unique hashes (such as `authorized_keys`), the researchers uncovered multiple web shell instances, including four zero-day malware samples with zero hits on VirusTotal.

In the second case study on Ivanti Endpoint Mobile Manager (EPMM), attackers leveraged a two-vulnerability chain allowing unauthenticated RCE just days after disclosure and public PoC release. By analyzing long-tail SHA-1 hashes linked to unusually large file sizes, the team identified two Sliver C2 framework implants (an open-source Go-based C2), alongside MySQL dumps, reverse shells, and web shells. Pivoting across shared IOCs, malware traits, and DNS history revealed that a single state-sponsored threat actor was responsible for both the PAN-OS and Ivanti campaigns.

### Takeaways Out-of-band disk snapshot analysis provides scalable forensic visibility into otherwise opaque cloud appliances. However, the speakers noted limitations during Q&A: snapshots cannot detect in-memory/fileless malware and will fail if the appliance disk is encrypted with vendor-managed keys. Defenders without broad cross-tenant data can still apply this technique by taking snapshots and diffing production instances against a clean, stock deployment baseline.