Adventures in the dark web of government data
Mining US federal government databases, especially FCC data, for geolocation and infrastructure discovery.
Summary
Government Data OSINT
The US federal government produces tens of thousands of forms collecting data on oil wells, imported vehicles, cell tower locations, and more. Most data remains locked behind outdated search interfaces or database formats.
Database Coverage
Scrapers and parsers work with hundreds of databases containing OSINT treasures outside Google and LexisNexis indexes. Coaxing OSINT from messy, hard-to-find data requires specialized techniques.
FCC Data Deep Dive
Over 20 million FCC licenses for regulated spectrum transmission leave data residue useful for geolocation of border surveillance infrastructure and McDonald's drive-thru radio frequencies.
Data Combination
Combining transmission protocols with contextual public data enables investigation. Automated Identification System signals on cargo ships joined with shipping records reveal what ships carry.
Related talks
Recon Village·27 min
xillwillx
Low-cost OSINT tool using web scraping to gather PII and passive information on targets without paid services.
Main Stage·36 min
Michael Schrenk
Michael Schrenk explores introduction to writing spiders and web agents at DEF CON 10, examining web-scraping and practical defenses.
Main Stage·41 min
Adam Bresson
Adam Bresson examines description from DEF CON 9 Speaker’s Page: Data Mining with PHP Adam Bresson has been programming in PHP, MySQL and HTML for over five years.