The Value Beyond the Hype: Applying Machine Learning in APT Detection

Advanced Persistent Threats are the plague of enterprise security, crippling businesses and IT careers through costly data breaches and massive losses. They routinely evade traditional security systems and steal confidential corporate information without a trace.

We have been fighting this enemy for years, sometimes not realizing who or what we are up against. It happened with Project Sauron, the sophisticated threat taking advantage of zero-day exploits to steal data from Internet-segregated computers of around 30 organizations worldwide in government, telecommunications, financial services and elite research facilities.

Bitdefender detects components of this omnipotent and unpredictable malware, with the help of machine learning algorithms embedded in its products for seven years now. But because systems are in charge of analyzing thousands of files daily, it’s sometimes hard to grasp the magnitude of the threat.

More algorithms, more accuracy

Targeted malware is constructed to avoid being detected by the security solution protecting the targeted system. Being so complex, it usually displays different behaviors found on several malware samples. So, to secure a high detection rate, security vendors prefer correlating the results of multiple algorithms. This strategy is effective in dealing with APTs, as opposed to using a single technology, because targeted malware’s attempts to avoid detection becomes significantly more difficult.

Bitdefender has been training algorithms for years - Perceptrons, Neural Networks, Centroids, Binary Decision Tree and Deep Learning, to name a few. Some are specialized on specific malware families, some on new malicious files and some are built to minimize the number of false positives. They complement each other, as well as traditional heuristic and signature-based detection.

We have realized that a single technology won’t solve all problems, but several technologies, together, can solve the majority of problems. Working as an ensemble, they provide a high rate of detection of new, unseen threats.

Detecting suspicious network traffic

Detecting zero-days starts with the assumption that the activity on an endpoint in an enterprise environment has a predictable behavior. So identification of any suspicious activity should begin with an analysis of typical network behavior.

Machine learning can help process millions of data points every minute to establish a “normal” baseline, comparing data points to past behavior, and identifying anomalous differences in values, over time. Using this technology, transactions, server processes, internet traffic, IPS alerts and proxy logs can be analyzed for unusual activities.

Neural networks to filter URLs

Malware has the ability to hide in a multiple-layered proxy network and to use evasion techniques capable of providing new binaries at every access.

One solution to block an APT attack is to use a network-based filter. Since a malicious URL is often accessible only for a couple of minutes, this method is only efficient if associated to a cloud-based service, as to update the list of domains and malicious URLs into everyone's computers as fast as possible.

Furthermore, sophisticated evasion techniques have reduced the efficiency of classic URL blacklisting-whitelisting, creating the need for more complex, dynamic techniques to block malicious attacks on the Internet.

Using unsupervised machine learning techniques, security experts can cluster URLs or domains to identify DGAs (domain generation algorithms), used by malware creators to generate domains that act as rendezvous points with the command and control servers.

Clustering partitions a data set into homogeneous groups based on given features. It seeks to structure unlabeled volumes of data in groups with the properties that all the objects within a group share. Clustering algorithms have to be fast (suitable for big data sets), generic (to be able to overcome obfuscation), as well as specific (a cluster must contain similar samples).

We have seen that using a hash-based clustering system on a large data set is an efficient way to filter malicious URLs. For every URL, a special hash is computed using a URL fingerprint. When an URL is being analyzed, the system will try to classify it using different algorithms, each with its own rate, resulting in two groups of algorithms (the ones that label the URL clean and the ones that label it as malicious). Then, the clustering system decides whether the fingerprint is malicious or not, using a vote system. The scores obtained for each fingerprint function are summed up to obtain the final score and, based on the results, the URL is signaled as clean or malicious.

Sorting email spam

APTs often take advantage of zero-day exploits in frequently used applications. Most of the time, the user has to visit a malicious page or open an infected document sent via e-mail. So, identifying malware-infected emails is still key in cyber-defense.

A trained machine learning algorithm can decide whether emails are spam based on rules it learns from pre-classified samples.

To quickly and accurately identify spam, Bitdefender uses local and cloud-based detection techniques to analyze SMTP connection information (sender IP address/sender domain), email header information, content information (text fingerprints, URLs, phone numbers, images, attachments etc.). The filters used for analyzing this data interact with each other to cover various scenarios and variants of already-detected spam waves.

Identifying malicious files

Sometimes, the payload is delivered on a device via PDF files embedded with malicious JavaScript. This way the code can be executed on computers while avoiding different protection methods. JavaScript is perfect as it is also polymorphic (two files with the same functionality may look very different).

To identify infected PDF files, we can use a clustering method based on the tokenization of the JavaScript code inside PDF files. By tokenizing the JavaScript code, describing it in an abstract manner and eliminating different operators used in polymorphism, we are able to obtain classes of files which are very similar syntax-wise. These can be easily clustered to reduce volume of work performed by human analysts.

Over time attack vectors change a lot, and our technology adapts constantly. This technology is now available for all kinds of script files, from PDF files with embedded Javascript, to Office documents, VBScript, AutoIt script files and standalone Javascript files.

• •