Analyzing the 2022 MITRE ATT&CK Evaluation for Managed Services

On November 9, the results of the first MITRE Engenuity ATT&CK® Evaluation for Managed Services were released. It evaluated the capabilities of 16 participating vendors in analyzing and detecting the behavior of an adversary run by the MITRE red-team. The test emulated activity based on the OilRig threat actor, and contextual output provided by the participants were mapped to the MITRE ATT&CK knowledge base. 

Bitdefender detected 100% of the attack steps while providing actionable, summarized output with a clear timeline of the attack and recommended actions.  

If you are considering a managed security service, whether a Managed Service Provider (MSP) or Managed Detection and Response (MDR), the results of the MITRE Managed Services evaluation are very informative. Evaluation of services is a subjective matter, and what may be a fit for one organization might not work for another. Results from this evaluation support a variety of use cases, and information provided by MITRE can help you identify the right security vendor for your specific needs.

About this evaluation

For this inaugural MITRE Managed Services evaluation, each vendor installed their preferred stack of security tools, some native, while others went with a compilation of tools. Bitdefender MDR leveraged our own native technology stack that serves as the cornerstone of our security portfolio. Our approach for this evaluation was to make it as close as possible to the real-world experience that our customers can expect. 

Remediation/prevention was prohibited in this evaluation, and the focus was entirely on understanding and reporting adversary activity. In an actual customer environment, Bitdefender’s security tools would not allow the attack scenario to proceed to later stages because of our extensive prevention and detection capabilities. For evaluations of the effectiveness of our prevention and detection capabilities, you can review results from the previous round of MITRE ATT&CK Evaluations, and results by an independent team from AV-Comparatives.

Analyzing the evaluation results

According to MITRE, a key aspect of the new MITRE Evaluation for Managed Services is that vendors are not expected to detect all techniques. The level and depth of provided information depend on the specific use case, and the first step in the evaluation of the results is to identify the problem you are trying to solve. 

Questions to ask include: Do you plan to use managed security services to cover most of your security operations instead of building your own Security Operations Center (SOC)? Are you looking to augment your internal resources? Or do you have an expert forensic team that is looking for additional sources of data? 

While a high volume of raw data is important for the scenario where a security team is seeking more, it’s counterproductive when you are trying to maximize the signal-to-noise ratio. Most vendors provided something for each step of the attack, and the complete communication and output are included in the report under option Download OilRig Archive. The archive for each participant contains two sections: “Email” with all email communication from the security team of a participant, and “Reports and Other Content” with any additional information and reports.

Information contained in these emails and reports is also visualized with a simplified overview of reported steps and sub-steps.  

Bitdefender results

Bitdefender detected 100% of the attack steps while providing actionable, summarized output with a clear timeline of the attack and recommended actions. The MITRE team generated a single incident along with some background noise over the course of 5 days, and we are proud of achieving a very high signal-to-noise ratio.  

For the duration of the attack, our team sent a daily incident report (typically 2-3 pages), which included: 

• Summary section 
• Details 
• Recommendations 
• Timeline 
• GravityZone incident IDs (for detailed technical analysis) 

We added a section called “Test Outcome Differences” for the MITRE evaluation. In this section, our analysts documented any changes compared to a real-world scenario where preventative actions would most often be enabled. For example, malicious binaries that would have been quarantined by our analysts, or endpoints that would have been isolated. 

In addition to daily summary emails, we also included the following documents in our communication: 

• Flash report (1-page) – A short notification when the active security incident was confirmed by our security analysts 

• After Action Report (9 pages) – A summary of the complete incident, with key points, summary, recommendations, and additional insights 
• Additional Info (17 pages) – The event timeline and screenshots from the GravityZone console 
• Sandbox reports – A comprehensive analysis of the detected malicious binaries
  

An example of communication with clear instructions when malicious activity is detected. Additional details are provided as an attachment 

The Bitdefender MDR team leverages our advanced (and native) security stack and adds human curation to cut through the noise and act early with an immediate response when a customer is experiencing a significant incident. Custom, approved response actions are tailored for each customer to ensure effective incident response while minimizing the business interruption risk. With pre-approved actions, we minimize the threat actor’s dwell time on your network using isolation, containment, and remediation actions. 

Conclusion

This was the first MITRE Managed Services evaluation. The testing is both effective and objective in its evaluation of what a managed service delivers in response to an emulated advanced attack. When considering services, your organization can benefit from delving into the data provided by this evaluation of the technical capabilities of a managed service.  

Recognize that generating data in response to an incident is a good initial step. Understanding how to relate data to the steps in a kill chain is better. Analyzing information within the context of your environment and the ATT&CK framework to advance your security posture is best.  

Each of those stages – data generation, information distillation, and contextual application - is geometrically more difficult. It gets simpler, though, when smart humans are involved in providing the service.  

As MITRE clearly explains,  

What we bring to the table is unique. Our real-world threat inspired methodologies are open and transparent. All results are publicly available and collaboratively produced with participants. There is no competitive analysis. We don't rank products against each other. And there is no “winner.” Instead, we show how each vendor approaches threat detection through the language and structure of the MITRE ATT&CK® knowledge base, and provide tools to allow the community to assess which product best fits their individual needs. 

You have requirements that are specific to your organization because your organization is unique. Bitdefender creates a baseline of every organization we work with so we can identify and analyze abnormal activity to determine if it is legitimate or not. As a security solutions and services provider, we know that your security teams need to focus on what is strategically imperative. We understand that the desired outcome of using a managed service, whether through an MSP, MSSP, MDR, or by leveraging EDR or XDR solutions, is to improve security outcomes by reducing the time your teams spend on distractions. 

Find out more about Bitdefender Managed Detection and Response on our MDR page.