Do's And Don'ts In File Formats

The talk looks at file formats through historical examples and argues that many were designed without enough structure, and that those early choices still cause problems. It opens with early word processors that saved whole blocks of memory instead of a structured file, then moves to MP3, which began as a raw stream of audio frames with no header or metadata. Players had to guess where data started by scanning for frame sync bytes, which makes small files ambiguous and lets a JPEG be misread as audio.

Over time MP3 gained footers, hardcoded length fields and a second header, so the same data ended up stored more than once. Duplicated fields are a security issue because one parser may check one copy while another acts on the other. The speaker contrasts this with a newer image format that has a fixed header and no room for metadata, and with older formats that put song titles first, which breaks scanning because the start of a file no longer reveals its type.

The practical advice is to start every file with a reliable magic signature, leave room for extensible metadata, avoid duplicate information, and avoid hardcoding values that were meant to vary, since parsers will treat them as constants. When a new format is needed, wrapping it in a well-known container such as the RIFF structure used by WAV and WebP lets designers reuse existing parsers and metadata handling. PNG is suggested as a reasonable model for a new format. The talk closes with audience questions.