Understanding File Type Identifiers & Scanners

The talk starts from a practical problem: when filtering very large volumes of files, you need to know what each file really is, and existing identification engines often disagree or fail. The speaker, who spent a year working with file formats from many communities beyond security, surveys the landscape. The tour covers libmagic and its derivatives, TrID, Detect It Easy, the Library of Congress format registry, PRONOM, Droid and the shared-MIME database. The engines differ in whether they rely on fixed byte sequences, follow pointers, use conditional logic, scan whole files or only the start, and in whether they use machine learning.

The middle of the talk looks at hard cases. PE executables can place their real header past a DOS stub, old Office documents change layout depending on their version, and JPEG identification depends on optional strings. Several engines cannot follow pointers or handle variable layouts, so they miss legitimate files. The speaker then presents Magika, a non-generative machine learning identifier that examines only the beginning and end of a file, returns probabilistic results in milliseconds, and is used in production. Its limits are also shown: it was not designed to resist adversarial or polyglot files, and it can miss unusual samples that resemble nothing in its training data.

The closing section covers evasion. Fast scanners can be fooled by manipulating headers, placing signatures outside the scanned range, or adding decoy signatures. A single-letter signature from an obsolete format produced a very large number of false positives on VirusTotal. The takeaway is that no engine is complete, so the choice depends on whether speed or depth matters most, and that legacy signatures can quietly cause false detections. The Q&A covered training data scarcity and attributing the software that produced a file.