PyFET: Forensically Equivalent Transformation for Python Binary Decompilation

### Overview PyFET presents a novel technique for improving Python binary decompilation through Forensically Equivalent Transformations (FET). Developed via a joint collaboration between the University of Virginia, Georgia Tech, and the University of Tennessee Knoxville, PyFET addresses the problem of decompilation failures in Python malware binaries compiled to bytecode. Decompilers often fail or produce incorrect output when processing complex Python binaries. Instead of modifying or patching complex decompiler codebases, PyFET directly modifies and transforms the binary bytecode beforehand to make it compatible with existing decompilers.

### Key Topics Decompilers encounter two primary categories of failures when processing Python binaries: - **Explicit Errors:** Failures where the decompiler raises an error message, often providing hints such as approximate bytecode offsets or function names. - **Implicit Errors:** Silent failures where decompilation finishes without warning but introduces semantic or structural corruptions. In a controlled study of 3,000 source files from PyPI and other repositories, researchers observed severe implicit bugs, including code coupled into incorrect scopes, unexpected `else` blocks injected into `try-except` statements, and while-loops inadvertently replaced with if-conditions.

To identify implicit errors, the authors developed a pattern-matching and mutation-based validation method. When a suspected pattern matches, the code is mutated with a corrected snippet, recompiled, and checked against the original binary at the target offset. This technique uncovered more than 22,000 previously undetected implicit errors across five decompilers.

### Technical Details To fix binaries without triggering side effects in decompilers, PyFET introduces Forensically Equivalent Transformations (FETs), extending semantically equivalent transformations (SETs). When strict semantic equivalence fails to prevent decompiler crashes, FET relaxes semantics while preserving critical forensic context: - **Transformation Rules:** PyFET utilizes 30 transformation rules targeting problem areas such as shortening lengthy conditional statements, dividing logical expressions, converting lambdas into standard functions, simplifying consecutive control flow, and moving statements (e.g., relocating a `return` statement outside a `with` block while capturing the result in a dedicated `fet_return` variable). - **Iterative CFG Optimization:** Rather than transforming entire binaries indiscriminately, PyFET inspects the control flow graph (CFG). It starts at the failure node, applies applicable transformation rules, and progressively expands to adjacent nodes along connecting edges until decompilation succeeds or all paths are explored. This strategy limits auditing to approximately 33% of nodes. - **Evaluation:** In an evaluation involving over 35,000 malware samples provided by ReversingLabs, 44.6% failed baseline decompilation, with failure rates rising on later Python versions. PyFET resolved more than 17,000 errors across five decompilers and successfully addressed binaries protected by obfuscators such as PyArmor/PJ Orion and opcode remapping.

### Takeaways Attempting to build a single "perfect" Python decompiler or patch disparate decompiler engines creates significant maintenance overhead and regression risks. Transforming Python bytecode directly via forensically equivalent rules provides an effective, scalable pre-processing pipeline that dramatically boosts decompilation success across diverse decompilers and heavily obfuscated real-world malware.