Detecting malicious behaviour using system calls

Detecting malicious behaviour using system calls

Van Mieghem, V.

Verwer, S.E. (mentor)
Doerr, C. (mentor)

Electrical Engineering, Mathematics and Computer Science

Intelligent Systems

2016-07-14

The emergence of Apple’s Macintosh computers’ popularity introduces new threats and challenges for the security on the Mac. For a long time, OS X security has benefitted from the popularity of Microsoft Windows. The threat landscape for the Mac is rapidly changing as the marketshare of the Mac is approaching 15%1. Malware on Apple’s OS X systems emerges to be an increasing security threat that is currently solely countered with ancient anti-virus (AV) technologies [18]. Current AV technologies pose a performance overhead on the entire system and have an inherent delayed effectiveness, due to their signature based detection [15][31]. In addition, current malware uses many forms of obfuscation to prevent detection by AV technologies, redering AV technologies useless against advanced threats [15][31]. Consequently, the need for more advanced detection and prevention techniques of malware is increasing. Detection of malicious behaviour instead of malicious signatures, ought to provide a more advanced form of protection. A system call is referred to as the request and service of specific, basic, functionality provided to applications by the operating system. This Master thesis answers the research question: “Is it possible to detect malicious behaviour per- formed by malware, based on monitoring system calls?” Presented is a novel, generic, behavioural detection and prevention mechanism for malware on OS X based on system calls. System call traces can be used to describe the behaviour of processes [11]. Much effort was put into the development of a kernel module that bypasses kernel security mechanisms and rewires one of the operating system’s core functionalities; system call handling. The rewiring of system call handling provided the ability to log all of the system call invocations performed by processes running on the monitored system. A significant amount of OS X malware and benign applications were executed in a monitored environment of which system call traces were collected. Based on analysing heat map visualisations and manual sequential analysis of the system call traces of both malicious and benign processes, anomalies in the malicious traces could be observed. Subsequently, several mali- cious system call patterns and detection rules were extracted providing detection of malware on OS X. The most successful defined pattern is constructed around the executions of Unix shell processes per- formed by malware. It is shown that this detection pattern results in a 100% detection rate of all malware possible to obtain for this thesis. Even advanced malware in an infected OS X application, known as OSX.KeyRanger.A, was detected using this method. In order to evaluate the False Positive Rate (FPR) accurately in real world scenarios, three different user profiles were defined. Applications distributed via the Mac App Store do not generate false positives. In case of the developer user profile type, the FPR increases to 20%. Applications responsible for the false positives feature a cross-platform nature, such as MATLAB, R, LaTeX and interpreters for scripting languages. A conducted survey under Mac users verified these conclusions. However, the number of false positive generating benign applications is very limited and whitelisting solutions provided can reduce the FPR in this developer user profile.

http://resolver.tudelft.nl/uuid:c71c85bc-d742-449b-88e7-33e172392ec2

Student theses

master thesis

(c) 2016 Van Mieghem, V.