MITRE ATT&CK Evaluations 2023: Deciphering the Results

On September 20^th, the results of the 2023 MITRE Engenuity ATT&CK Evaluations for enterprise security solutions were released. This year, Bitdefender detected 100% of the attack steps, while providing the highest possible level of description of the simulated adversary behavior for each step.

This is the third consecutive year participating in the MITRE Engenuity ATT&CK Evaluations that Bitdefender has achieved this result. This year, 31 vendors participated in the evaluation, which assessed the product’s ability to detect and protect against emulated adversary tactics and techniques. Round 5 of the MITRE ATT&CK evaluations focused on determining security product capabilities against adversary behavior inspired by Turla, a sophisticated Russian-based threat group that has infected victims in over 45 countries.

This article offers tools to thoroughly examine the data. Combined with other information, these insights will help you understand and use the results effectively for your organization's needs. This includes examining the importance of ATT&CK Evaluations to gain insights into vendor’s true detection capabilities.

Why MITRE Engenuity ATT&CK Evaluations Matter

ATT&CK Evaluations have grown in popularity as an important source of objective insights into the capabilities of security products and they complement other trusted independent tests from organizations such as AV-Comparatives and AV-TEST.

Uniquely among similar organizations, MITRE Engenuity leverages their expertise and ATT&CK knowledge base to build a complex and rigorous attack emulation inspired by tactics and techniques used by threat groups. For the detection test, a products’ ability to block malicious activity is deactivated allowing the MITRE team to focus on determining detection capabilities, including the level of details or context provided by the security tools. Unlike other independent security trials, ATT&CK Evaluations results do not declare a “winner” or rank vendors. This is because, as stated on the MITRE website, “there is no singular way for analyzing, ranking, or rating solutions,” rather these evaluations reflect each vendor's unique approach to threat detections.

Deciphering MITRE ATT&CK Results

While there is a wealth of information which may look confusing at first glance, the ATT&CK Evaluations are a rich source of objective insights which everyone should mine. The most important interpretation of the results is from the perspective of the needs of your organization. While Bitdefender detected 100% of the attack steps, there may be sub-steps (and there are many) which your organization values more than aggregated steps.

For example, there is useful guidance from MITRE on leveraging the results to evaluate products. We will quote some of these recommendations along with our own analysis of this year’s Turla results.

This year the MITRE team evaluated the detection capability of each enterprise security product across 19 major steps and 143 sub-steps that emulated observed tactics and techniques of the notorious Turla group.

To provide a classification of different types of detections, MITRE ATT&CK used detection categories with each building on the previous and including more context:

• N/A
• None
• Telemetry (minimally processed data showing events occurred)
• General (processed data specifying that a malicious/abnormal event occurred)
• Tactic (correctly identifies the goal the adversary was trying to achieve)
• Technique (also identifies the way the action was performed)

For each sub-step of the emulation, the highest level of detection is recorded per product.

Image Source

To explore the results on the MITRE website, you can compare vendor’s detections across individual techniques or sub-steps or look at high-level counts for analytical coverage (includes general, tactic, or technique detections), telemetry coverage, and visibility (telemetry or analytical coverage).

Choosing the right metrics that best align with your needs is influenced by factors like your team's expertise, available time and resources, and the acceptable level of noise.

The number of sub-steps in the 2023 MITRE Evaluations has increased from 109 to 143 with some techniques being more impactful or relevant in revealing a potential attack than others. This demonstrates the limitations of counting detections indiscriminately. According to MITRE, “you might assume a higher analytic count is better, but a high number might indicate the possibility of flooding your analyst with false positives or potentially redundant alerts.”

From our perspective, a primary metric relevant for most organizations is detections on major attack steps with technique-level context. Detections of relevant techniques across all of these steps allows teams to stop an attack. The next, and deeper level of analysis of the results this year looks into product capabilities when it comes to detecting specific techniques or sub-steps that are important and relevant to your team.

Bitdefender's Results: Clear Insights with Minimal Noise and Complexity

Bitdefender GravityZone has achieved 100% detection of the 19 attack steps in the fifth round of the MITRE Engenuity ATT&CK Evaluations for enterprise security, offering technique details on each of the steps that would enable SOC teams to uncover and stop the attack. This is the third consecutive year that Bitdefender can detect and provide actionable insights on each of the attack steps in the ATT&CK Evaluations.

Upon examining the results, we achieved analytical detections for the sub-steps we aimed to cover, avoiding to alert on those that might create unnecessary noise for security teams.

To better understand the results and decide which vendor’s solution is a better fit for your team, we recommend that you compare vendor results on specific sub-steps while considering which are important to you and which you would not want to be alerted on.

The Bitdefender approach to endpoint detection and response (EDR) ) and extended detection and response (XDR) is centered on enabling analysts, regardless of their team size and level of expertise, to identify and respond swiftly and accurately to attacks. The first step to achieve this is reducing alert fatigue by presenting high-fidelity detections and automatically triaged and correlated incidents that minimize noise and reduce the chance of real threats getting lost in an ocean of irrelevant alerts. Actionable context presented in a human-readable way, and simple response recommendations further aid even under-resourced teams to respond fast while capabilities such as Live Search support advanced threat hunting.

Conclusions and Recommendations

As many organizations struggle with alert fatigue, a skills gap, a lack of visibility, or inefficient SecOps workflows, MITRE ATT&CK Evaluations remain a key source of objective and transparent insights into the capabilities of different detection and response tools.

MITRE avoids establishing rankings since they don’t reflect the unique challenges and needs of each organization. Instead, every organization needs to explore the nuance of ATT&CK Evaluations results to understand how different approaches are likely to behave in a real environment. Exploring these results can be a useful step in deciding which solutions are candidates in solving the problems your team encounters.

While understanding the results of MITRE ATT&CK this year, it’s important to zoom out and look for vendors which consistently perform well in ATT&CK Evaluations, year after year, and to look at other trusted independent evaluations which compliment MITRE, such as AV-Comparatives and AV-TEST. These evaluations highlight important aspects which are not covered in MITRE such as: early prevention or protection efficacy, false positive rates, performance impact, usability, and Return on Investment (ROI).