Mohamad Mokbel - Crawlector: A Threat Hunting Framework

### Overview Mohamad Mokbel presents Crawlector, a threat hunting framework created by combining the concepts of a crawler and a detector. Developed as an experimental C++ project, Crawlector scans websites to identify malicious web objects, compromised sites, watering-hole attacks, and exploit kits (such as drive-by downloads). It takes lists of domains and YARA rules as input, spiders and crawls target pages, evaluates the retrieved content against rules, and outputs structured detection results.

### Key Topics - **Motivation and Architecture**: The author built the framework out of curiosity and a preference for C++, avoiding Python and high-level indentation-based languages. Crawlector compiles into a single executable for Windows with no external dependencies and is entirely controlled via an INI configuration file (`CL_config.ini`) rather than command-line arguments. - **Spidering and Crawling**: Crawlector distinguishes between spidering (discovering additional links within a page up to a hardcoded depth limit of two levels for performance) and crawling (scanning input lists). It provides granular filtering to target specific contexts, such as checkout or product pages for detecting web skimmers, or isolating external links/iframes. - **Data Collection and Telemetry**: The framework supports both online and offline scanning, page caching, domain IP resolution (IPv4/IPv6), and proxy-aware HTTPS communication. It collects digital certificates and inspects statistical attributes (issuers, subject details, key lengths), and generates standard symmetric hashes (MD5, SHA-256, RIPEMD) as well as Trend Micro's fuzzy hash TLSH for clustering similar web pages. - **Threat Intelligence Integration**: Crawlector integrates with abuse.ch’s URLhaus database (either live via queries or offline via disk) to cross-reference URLs against over 113 million malicious entries, retrieving context such as malware family and campaign phase. - **Rule Transpilation**: Because Crawlector relies on YARA's C API for static rule evaluation, Mokbel created an automated transpiler to convert public rule sets from Fiddler's EK Fiddle plugin (the `master-regexes` file) into compliant YARA rules, adjusting for YARA's regex engine constraints.

### Technical Details - **Input Formats**: Accepts domains in three formats: Type 1 (plain domain per line), Type 2 (custom domain IDs), and Type 3 (spidering directives configuring `total` links, `depth` up to 2, and `sleep` delay in milliseconds between requests). - **Output and Storage**: Saves page bodies, extracted certificates, and discovered links into dedicated folder structures, logging scan detections and metadata into CSV format (with a recommendation to convert to SQL for querying). - **Limitations**: The framework is single-threaded in its initial release and relies purely on static pattern matching without dynamic JavaScript evaluation or headless browser integration. Digital certificate scanning is currently mutually exclusive with other scanning features.

### Experimental Findings Mokbel evaluated Crawlector across large real-world domain sets, including 700,000 top Alexa sites, 100,000 WordPress sites, and 127,000 Magento sites sourced via BuiltWith: - **Magento Scan**: Scanning active Magento websites took approximately 10 days, processing 104,000 active domains. 20 distinct YARA rules triggered across 139 malicious sites with only a single recorded false positive. - **Threats Identified**: Scans detected diverse compromises, including exposed PHP web shells, malicious redirect chains, coin-mining scripts (Coinhive), and widespread Magecart/hex-encoded web skimmers. - **Evasion and Obfuscation**: Detections revealed malicious JavaScript exfiltrating credit card data over secure WebSockets (WSS) and steganographic skimmers hiding code inside PNG or GIF images loaded via `.slice()` calls. - **Clustering**: Body hashing on 104,000 Magento pages yielded only 1,328 unique hashes, demonstrating that content clustering significantly reduces the surface of unique templates needing review.