Dark Web Digger: Modular Scraping for Dark Web Intel

The talk starts with the problem of anonymous users on dark web forums, who keep several aliases and move between sites as forums go offline and return. Tracking which accounts belong to the same person, and which topics are gaining traction, is slow to do by hand. The student presenters, working with a university adviser and a sponsoring cyber insurance company, built a prototype scraper split into independent parts: a collector, a parser, a database layer, and a configuration layer that ties them together.

The collection path runs through the Tor browser behind a VPN, on a Linux server, using Python with Selenium, Beautiful Soup, pandas and MySQL. Logging in still needs a manual step because of captchas, which the team has not been able to automate. Once running, the tool saves forum pages, individual posts and author profiles, writes them to CSV output and a database, and produces per-user files. Config files in TOML let the team swap browsers and target sites without rewriting code. Questions from the audience covered cost, open-sourcing, and whether the approach would hold up against deliberate stylistic disguise.

Next steps involve machine learning: semantic similarity and linguistic profiling to match writing style across aliases, trend detection across users and forums, and a retrieval-augmented chatbot that analysts could query for posts and activity. The presenter stressed that matching would be probabilistic rather than foolproof. The practical takeaways are to design scrapers as swappable modules for volatile sources, and to plan for captcha handling as the main obstacle to full automation.