Decoding the MITRE Engenuity ATT&CK® EVALUATIONS 2021 Results

On April 20^th, the results of the latest round of the annual MITRE ATT&CK® Evaluation of security solutions were released. This year, a field of 29 security solutions from leading cybersecurity companies including Bitdefender, Crowdstrike, and Microsoft were tested on their ability to detect the techniques and tactics of Carbanak and FIN7.

These notorious cybercriminal groups have wreaked havoc by compromising financial services and hospitality organizations using sophisticated malware and techniques. Carbanak has generated more than $300 million in damages (to date) for hundreds of banks across 30 countries, while FIN7 has exfiltrated more than 15 million credit card records from victims worldwide. (More details on Carbanak’s operations are available in a dedicated Bitdefender research paper.)

 

What makes MITRE ATT&CK Evaluations unique and highly valuable?

MITRE ATT&CK Evaluations leverage methodology unique among cybersecurity industry tests. Instead of simply testing the solution’s ability to detect and block cyber threats, MITRE meticulously emulates the full behavior of sophisticated attacks (they did this for APT3 or APT29 in previous rounds and Carbanak/FIN7 in this most recent round). MITRE requires blocking features of the tested products to be disabled during the evaluation. This approach reveals in detail the capabilities of the various technology layers embedded in the solutions to detect, analyze, provide telemetry and visibility on all phases/sub-phases of the attack kill-chain.

The extensive MITRE ATT&CK knowledge base is instrumental for building the testing methodology as it provides a common vocabulary and alignment throughout the cybersecurity industry. The value of the MITRE ATT&CK evaluation is in the ability to analyze the robustness and completeness of the tested solutions. This makes the test highly relevant for organizations interested not only in the ability to automatically block attacks but also those looking to fight advanced attacks throughout all execution phases. Using MITRE’s approach in security operations greatly enhances the cyber resilience of the organization and the chances of cyber defenders to detect and gather valuable insights for adversary activities.

 

How can you use the results to guide your cybersecurity decisions?

An unusual characteristic of the MITRE evaluations is the lack of competitive rankings, allowing various interpretations that make it more difficult to navigate the results. This shouldn’t discourage the cybersecurity community from investigating the analysis because effort spent understanding the data will pay dividends. The results provide an in-depth understanding of the tested solutions’ behavior and represent an excellent complement to other industry tests.

To begin with an obvious but significant point, there is not a single “right perspective” to look at the MITRE ATT&CK Evaluations results. The evaluators do provide a few hints on how to interpret the data best. The key metrics used for presenting the data are:

• Detections — any information, raw or processed that can be used to identify adversary behavior. This metric includes raw telemetry (like Process Start or File Create) and analytic detections (like General Detection, Tactics, or Attack Techniques). The Detection Count represents the sum of detections of all types. Note that any of the 174 sub-steps included in the attack can have more than 1 detection (thus, the total number of Detections can exceed the number of sub-steps).
• Telemetry Coverage — the number of sub-steps where telemetry was available.
• Analytic — any processed detection, such as a rule or logic applied to telemetry (e.g., ATT&CK technique mappings or alert descriptions).
• Analytic Coverage — the number of sub-steps where 1 or more analytics were available.
• Visibility — the number of sub-steps where an analytic or telemetry was available.

These key metrics are available for each of the 29 participating vendors by visiting the vendor dedicated results page.

 

Which metrics are relevant for your organization?

Next, you will want to understand which metrics are most relevant for you. This will depend on your organization’s profile. The Detections are obviously relevant for any organization, as the more elements of the attack the solution is capable of detecting, the more effective it is, generally speaking. Telemetry is a valuable metric in the context of an organization that has a security operations center (SOC) where tools, resources, and know-how exists for further analysis of the raw information. Compared to raw telemetry, the Analytics metrics provide context for the detections. Actionable analytics help reduce the risk of alert fatigue and the investigation effort required from security analysts. This makes Analytics a highly valuable metric for organizations with resource-constrained IT and security operations teams. Overall Visibility is a combination of raw detections (Telemetry Coverage) and contextualized alerts (Analytics Coverage) that provide a general view of the solution’s ability to provide visibility into the attack elements.

 

How to read Bitdefender’s results from the latest MITRE Engenuity ATT&CK^® EVALUATIONS

Bitdefender chose to participate in the MITRE ATT&CK evaluation for a second time because it is a trusted industry framework highly regarded by industry analysts and organizations looking for a reliable and effective detection and response cybersecurity solution. This year’s results re-confirm what the previous round also revealed: Bitdefender GravityZone has an exceptional ability to detect the attack steps and sub-steps, complemented with rich and actionable context for nearly all detected actions.

With a total count of 366 detections (Fig.1), Bitdefender achieved the highest number of detections of the 29 cybersecurity vendors who participated in the MITRE Engenuity ATT&CK Evaluations. This round showcased Bitdefender GravityZone as the leading solution for detecting the broadest range of cyber threats with nearly a 50% higher detection count than the average number of detections among the vendors evaluated.

Figure 1 

Bitdefender also stands out in results for enabling efficient security operations and reducing alert fatigue by providing analytics insights for 96% of all detections. Figure 2 displays how the sub-steps detections are complemented with General details and/or insights into the used Tactics or Techniques. By comparison, many other vendors evaluated generated high volumes of Telemetry without additional context, which necessitates deeper investigation by already burdened security teams.

 

Figure 2 

Through exceptional accuracy, context-rich alerts, and multi-OS support, including 100% detection of attack techniques for Linux systems, Bitdefender GravityZone demonstrated that it is a smart choice for organizations looking to enhance the cyber resilience of their heterogeneous environments with a unified cybersecurity platform that offers the highest rate of detection and actionable alerts.

Learn more about how to enhance cyber resilience with GravityZone.