Ambly: The Darknet Spider

In this Recon Village presentation, security researcher Levi introduces Ambly, an open-source web crawler and spider purpose-built for indexing and scraping hidden services on the Tor darknet network [00:00]. Levi outlines the unique architectural hurdles inherent to darknet reconnaissance, including high latency, unindexed hidden service directories, and the extreme volatility of onion sites that frequently appear and vanish without notice [07:23, 30:19].

Levi breaks down Ambly's modular software architecture, which interfaces with Tor through local SOCKS5 proxies to execute automated circuit rotation and distributed scraping [14:37, 22:13]. The crawler parses raw HTML, extracts referenced `.onion` URLs, strips metadata, and indexes text content into an isolated database while storing binary artifacts securely to protect analysts from malicious browser exploits [37:35]. He demonstrates how Ambly traverses community forums like Torum and Dread, following hyperlinks to uncover nested underground discussion boards and service listings [30:19].

The talk demonstrates how threat intelligence teams and red teams utilize Ambly to automate darknet surveillance. Levi showcases practical workflows for brand monitoring, tracking corporate credential leaks, identifying compromised employee access tokens, and mapping illicit marketplace supply chains across the dark web [37:35].

(AI summary — Gemini Flash)