#HITB2023HKT #COMMSEC D1 - How I Learned To Stop Reversing And Love The Strings

### Overview In this talk, Paolo (known online as babush) presents "Big Match," a research proof-of-concept developed to identify open-source libraries embedded inside compiled binaries by leveraging extracted strings. Reverse engineers frequently spend days analyzing proprietary binaries only to realize they are analyzing open-source code linked statically into the target. Because manual string searches using standard search engines or code search tools can be noisy, fail due to nested static libraries, or omit parent packages, Big Match automates binary identification by matching binary strings against a massive database of GitHub C/C++ repositories without relying on machine learning.

### Key Topics - **The Pitfalls of Manual Reverse Engineering:** Relying on ad-hoc string searching often misleads analysts when static linking nests one library inside an outer framework (for instance, identifying libsrtp within a VoIP binary while missing pjsip). - **Dataset Construction via GHTorrent:** Utilizing GHTorrent dumps (specifically MongoDB exports processed with PyMongo) to access GitHub repository metadata, stars, and commit histories without hitting clone bandwidth limits. - **Repository Deduplication Challenges:** Filtering forks and duplicate projects across thousands of repositories. The speaker discusses graph traversal edge cases, such as the `octocat/Spoon-Knife` tutorial repository being merged into the commit history of `LibreCAD`. - **Vector Space Information Retrieval:** Treating extracted string hashes as terms and repositories as documents using TF-IDF weighting and cosine similarity. - **Clustering Search Results:** Using spectral co-clustering to isolate distinct libraries present together in a single binary instead of returning redundant forks of the first match.

### Technical Details - **String Extraction & Filtering:** Repositories were scanned using `ripgrep`, followed by custom Python string de-escaping and hashing. Common boilerplate strings degraded matching accuracy; empirical analysis showed that pruning the top 10,000 most frequent strings (such as generic error messages) substantially improved results while preserving useful signatures like `GIF89a`. - **Data Pipeline:** The ETL pipeline was built largely using Bash scripts and standard Unix utilities, which provided surprisingly fast execution (building the search database in approximately one day). - **Secondary Deduplication:** To address excessive RAM consumption (averaging 40 KB of RAM per repo and 23,000 strings per repo), repos of similar zip file sizes were unpacked and deduplicated using Jaccard similarity thresholds, stripping thousands of Linux and Android forks. - **Search Execution:** In the web demo, client-side JavaScript extracts strings from uploaded binaries, hashes them, and submits them to the server. The engine runs a matrix multiplication of query vectors against the repository matrix, followed by spectral co-clustering to display top matching independent libraries.

### Takeaways - Exact string hash matching provides an effective, privacy-preserving, 0% machine learning mechanism to identify open-source components, as the server only receives string hashes rather than proprietary binary files. - The approach works reliably when target binaries contain distinctive strings and can be extended to magic numbers, version range detection, monorepo dependency audits, and malware classification. - Future directions include integrating string matching directly into decompiler plugins (such as rev.ng) to enable function-level string matching and automatic symbol renaming.