Hacking the EULA

Architecture and Background

Marce Luck & Tom Stracener: Hacking the EULA: Reverse Benchmarking Web Application Security Scanners Each year thousands of work hours are lost by security practitioners as time is spent sorting through web application security reports and separating out erroneous vulnerability data. Individuals must currently work through this process in a vacuum, as there is no publicly available information that is helpful. Restrictive EULAs (End User License Agreements) prohibit examining a signature code-base for common errors or signature flaws. Due to the latter point, a chilling effect and has discouraged public research into the common types of false positives that existing commercial technologies are prone to exhibit.

Vulnerability Analysis

Reverse Benchmarking is a new species of reverse engineering that involves running a security solution against an application designed to solicit false positives. Unlike testing scenarios that emphasize gathering valid or accurate data, Reverse Benchmarking involves exposing architectural or logical flaws within a web application scanner by employing techniques to trick simple rule-based mechanisms. Running a scanner against a Reverse Benchmark target quickly reveals faulty rules, flawed testing logic, or poorly written or implemented security testing procedures. Additionally, a Reverse Benchmarking application will expose patterns in the propensity of a scanner to report false results, making it easier to spot false positives when they occur in the future.

Exploitation and Demonstrations

Reverse Benchmarking opens up new opportunities for studying and improving existing web application security technology by exposing common faults in testing logic that are often the culprit of massive false positives. In turn this facilitates research into a taxonomy of general false positive types, ideally, a schema for mapping particular security tests to a common, generic language. This can provide a framework around which public discussion, research, and documentation of such flaws can occur without violating EULA agreements. We will also discuss the formation of a open community initiative centered around the use of Reverse Benchmarking to study false positive types."Marce has been working in the information security field since the late 90s and during that time has worked for a bunch of places once, but no place twice.

Remediation and Defensive Strategies

Marce's former employers include: the CERT/CC, IBM, Farmers Insurance, Deloitte, Cenzic, and Himself. He currently works for A Fortune 100 Financial Company as a Information Security Architect, and enjoys it very much. Tom Stracener is a Sr. Security Analyst for Cenzic's CIA Labs. At Cenzic Tom has played an important role in the evolution of their flagship technology Hailstorm, and was one of the chief designers of the Cenzic HARM Score, the Hailstorm Application Risk Metric. He usually embarrasses himself horribly at Defcon at least once a year, but this is his first time to do it before an audience. Tom has spoken at more than 50 conferences and events in the last 2 years.