Smashing the Stack with Hydra

Architecture and Background

Smashing the Stack with Hydra: The Many Heads of Advanced Polymorphic Shellcode Pratap Prabhu Research Assistant, Columbia University Yingbo Song Research Assistant, Columbia University Salvatore. J. Stolfo Professor of Computer Science, Columbia University Recent work on the analysis of polymorphic shellcode engines suggests that modern obfuscation methods would soon eliminate the usefulness of signature-based network intrusion detection methods and supports growing views that the new generation of shellcode cannot be accurately and efficiently represented by the string signatures which current IDS and AV scanners rely upon. In this presentation, we expand on this area of study by demonstrating never before seen concepts in advanced shellcode polymorphism with a proof-of-concept engine which we call Hydra. Hydra distinguishes itself by integrating an array of obfuscation techniques, such as recursive NOP sleds and multi-layer ciphering into one system while offering multiple improvements upon existing strategies.

Vulnerability Analysis

We also introduce never before seen attack methods such as byte-splicing statistical mimicry, safe-returns with forking shellcode and syscall-time-locking. Multi-tasking shellcode with safe-returns ensures that we bypass sensors that monitor application crashes. Also, we bypass online emulators by deriving an encryption key from the OS environment -- something that is not easy to implement in an emulator. In total, Hydra simultaneously attacks signature, statistical, disassembly, behavioral and emulation-based sensors, as well as frustrates offline forensics. This engine was developed to present an updated view of the frontier of modern polymorphic shellcode and provide an effective tool for evaluation of IDS systems, Cyber test ranges and other related security technologies.

Exploitation and Demonstrations

Pratap Prabhu is a Masters student at Columbia where he is working in the Intrusion Detection Systems lab. At Columbia, he has contributed to several projects including developing an advanced polymorphic engine. Prior to joining Columbia, he has had previous experience working as a technical researcher in an upcoming IDS company. Yingbo Song is currently a PhD student at Columbia University where he works with the Intrusion Detection Systems group and the Machine Learning group. Previously, he has worked on shellcode polymorphism and machine-learning-based sensors for web-layer code injection detection.

Remediation and Defensive Strategies

Salvatore J. Stolfo is Professor of Computer Science at Columbia University. He received his Ph.D. from NYU Courant Institute in 1979 and has been on the faculty of Columbia ever since. He has published extensively in the areas of parallel computing, AI knowledge-based systems, data mining and most recently computer security and intrusion detection systems (see www.cs.columbia.edu/ids). His research has been supported by DARPA, NSF,ONR, NSA, CIA, IARPA, DHS and numerous companies and state agencies over the years while at Columbia. For copies of the slides and additional materials please see the DEF CON 17 Archive here: