Doctoral Dissertations
Date of Award
8-2024
Degree Type
Dissertation
Degree Name
Doctor of Philosophy
Major
Data Science and Engineering
Major Professor
Jeff A. Nichols
Committee Members
Edmon Begoli, Amir Sadovnik, Catherine Schuman, Michael Jantz, Sean Oesch
Abstract
Machine learning allows for the detection of novel malware. However, this method
of detection introduces new vulnerabilities in the form of feature extraction evasion
and adversarial instrumentation. These emerging methods for evading detection are
hampered by the need to maintain functionality in altered binary files, a challenge
largely unique to this domain. Functionality preservation is necessary to maintain the
true label of altered files. Binary files, especially Windows Portable Executable files
with an X86/86-64 architecture, may contain bytes whose role in functionality can
only be determined via manual reverse-engineering. Therefore, automatic methods
for altering these files without loss of function are highly restricted; we advance
the state of research via the study of malware obfuscations and the development of
novel actions. First, we address feature-extraction evasion by producing detection
models and remediation tools for these methods. Second, we demonstrate that static
binary instrumentation methods can evade detection by commercial off-the-shelf tools
without guidance. Third, we present a wide-spectrum survey of existing methods
for guiding static binary instrumentation via adversarial machine learning. Finally,
we demonstrate a set of actions paired with AI guidance and verify the effect these
actions have on the evasion and functionality-preservation rate of altered binaries
using strict criteria.
Recommended Citation
Koch, Luke R., "EMBERs in the Dark: Countering AI-Based Malware Detection via Static Binary Instrumentation. " PhD diss., University of Tennessee, 2024.
https://trace.tennessee.edu/utk_graddiss/10473
Included in
Artificial Intelligence and Robotics Commons, Data Science Commons, Information Security Commons