The Essential Guide to Independent Cybersecurity Testing

Many businesses struggle with finding the right cybersecurity solutions and services to fit their needs. Helping to bring clarity to the situation, independent testing has emerged as a cornerstone for informed decision-making. The comprehensive analysis aims to unravel the layers of complexity that ultimately lead to finding the right vendor, but what factors should organizations consider when evaluating the results?

The Importance of Independent Testing

Independent evaluations stand in contrast to the often misleading, pay-for-play evaluations that populate the cybersecurity landscape. Unlike vendor-commissioned reviews, where results might be skewed in favor of the funding entity, independent tests offer an unbiased assessment of a product's efficacy. This impartiality is crucial; it ensures that organizations rely on verified data rather than persuasive marketing narratives. Vendor participation in these independent tests is also important, as it reflects the vendor’s willingness to be completely transparent about their solutions or services. By favoring solutions vetted through rigorous, unbiased testing, companies can make decisions based on merit and effectiveness, ensuring their defenses are as robust as possible.

The Role of AMTSO and Its Impact

In 2008, the Anti-Malware Testing Standards Organization (AMTSO) was founded by a group of academics, reviewers, testers, and vendors within the cybersecurity industry, aiming to improve the objectivity, quality, and relevance of malware testing methodologies and to provide guidelines for accurate and fair evaluations of anti-malware solutions. The AMTSO significantly influences the cybersecurity evaluation ecosystem. AMTSO's guidelines help in creating realistic testing environments that mimic actual attack scenarios, thereby providing insights into how security solutions perform under real-world conditions. This contribution is invaluable, as it elevates the quality and reliability of independent testing, offering organizations a trusted benchmark against which to measure cybersecurity solutions. Independent testing firms certified by the AMTSO are deemed more reliable sources of unbiased assessments, and among them are:

To become AMTSO-certified, testing labs undergo a rigorous evaluation process to ensure they meet the high standards set by the organization. Certification signifies that the evaluator has demonstrated a commitment to these principles, and this should give confidence to any organization that’s considering the results of one of these independent evaluators.

When evaluating the results of independent tests, there are several factors that organizations need to keep top-of-mind so that the result is making the right decision for their business.

Tailoring Test Results to Organizational Needs

Choosing a cybersecurity vendor is a complex decision influenced by several factors, with your organization's specific needs being the top priority. Start by streamlining your vendor list based on your unique requirements. This could include the specific operating systems that need protection or whether your network has air-gapped or offline segments.

Before evaluating test results, organization decision-makers have a few key areas they need to consider regardless of the size of their entities:

Organizational Infrastructure: When evaluating a solution, consider if it covers all your endpoints – workstations, servers, cloud workloads, IOT devices, legacy systems, OS types, remote workers, etc. Take inventory of your network structure and determine how much of a challenge it will be to deploy the solution across your organization. Further consideration is needed if your business requires managing environments with confined or air-gapped networks.
Compliance and Data Protection Requirements: Adhering to regulatory requirements is critical. Consider solutions that ensure data protection compliance across all jurisdictions your business operates in, such as GDPR, HIPAA, etc., and whether those regulatory requirements are met by the vendor.
Solution and Service Scalability: The solution and services the vendor offers must be able to scale to cover all your devices across your organization’s geographic locations.
Ease of Use and Management: Considerations must be made on the amount of training the staff will need to familiarize themselves with the vendor. A solution that is overly complex and cumbersome to use can lead to an increase in response time should a security incident occur. Centralized management and automation features can help streamline security operations for better outcomes.
Vendor Consistency: Prioritize vendors with a proven track record. Evaluating a specific test result can give you a snapshot as to how that vendor performed during a particular period, but due diligence should be done in reviewing how that vendor has performed in the past. Consistently good performance is a good indicator that the vendor will continue to effectively manage current and future threats.

These baseline considerations can help form the foundations for making an informed decision. For the actual test results, there are other critical capacities that need to be considered.

Understanding What is Being Evaluated

Before evaluating any independent testing results, one must be aware that there are differences between the evaluators. Some evaluators are focused only on specific markets or specific threat types. Some tests are skewed more for larger enterprise customers while some are tailored more towards small and medium size organizations. Many use a ranking system and provide awards to the best performers, but some do not. Some publish their results so that it can be easily consumed by the masses, with charts, graphs and figures most IT professionals can easily interpret, while others publish information tailored to experienced security analysts.

A good example of this is the MITRE ATT&CK® evaluations that are mostly focused on detection of advanced persistent threats (ATPs) commonly used by notorious threat actors. MITRE gives importance on how much detection detail a vendor can provide. For MITRE it’s not just important that the threat was detected, but reporting how that detection was made is essential. This type of specificity can be useful for organizations that have their own security operations center and want to review as much detail as possible on how a specific threat actor operates. In their evaluation, however, MITRE gives no attention to the product’s propensity for generating false-positives or the performance impact the solution may have on systems. MITRE® also does not use a ranking system, and the results are published in intricate tables that require some level of security expertise to understand. By contrast, factors such as false alarms and system performance impact are weighed heavily in test from the likes of AV-Comparatives and AV-Test. Reviewing the vendor’s capacity to protect against advanced threat is critical, but how that will impact every day usage should not be overlooked as this can affect productivity and threat-response efficiency. Every organization should take into consideration the factors of the evaluation that are most important for them and weigh those factors before making a final decision on a vendor.

How to Choose the Right Security

Given the sophistication and sheer volume of modern cyberattacks, it’s more important than ever to make the right choice in deciding on security vendor. All vendors offer some degree of protection against modern threats, but minor differences can make a huge impact during a real security incident. Different approaches can influence the threat’s impact, scope, time-to-resolution (TTR), and remediation capacity. Let’s review a few important points to consider:

Detection of Advanced Attack Types

Any solution prepared to tackle the sophistication and scope of modern cyberattacks cannot rely on legacy technology for threat detection. Solutions that rely solely on signature-based detection are antiquated and ineffective. Any security solution evaluated must include heuristic detection of threats using AI and machine-learning, something Bitdefender pioneered back in 2008. It should be able to monitor network activity and protect against advanced lateral-movement techniques, fileless attacks that avoid disk access, compromises of legitimate programs (living-off-the-land attacks), and other novel tactics and techniques.

Covering the Entire Kill Chain

A cybersecurity solution's proficiency in thwarting threats across the entire kill chain, from initial reconnaissance to data exfiltration, is critical. This comprehensive defense strategy ensures that even if attackers bypass initial barriers, subsequent layers of security are in place to thwart their progress. A solution that effectively addresses multiple stages of the kill chain provides a more resilient defense against complex, multi-vector attacks. The best protection involves shielding the largest attack surface area possible, and for many organizations visibility is key. Any activity by the threat actor that goes undetected can be costly.

Thwarting Attacks Before they can Act

Stopping threats at the pre-execution stage is significantly more effective than on-execution or post-execution interventions. Pre-execution security measures prevent malware from activating its malicious payload, thereby averting potential damage. This proactive stance not only reduces the risk of compromise but also minimizes the need for remediation efforts, which can be resource-intensive and costly. In essence, preempting threats before they execute is akin to stopping an intruder at the door, a far preferable scenario than dealing with the consequences once they're inside. AV-Comparative’s Advanced Threat Protection tests do a great job of measuring how well the evaluated vendor’s solution performs in this category. Highlighting the importance of stopping threats before they can perform any action, AV-Comparatives comments, “A good burglar alarm should go off as soon as someone breaks into your home. It should not wait until they start stealing.”

Assessing Performance Impact and Cost Considerations

The performance impact of a cybersecurity solution extends beyond mere productivity; it can also directly influence operational costs, especially in cloud-based environments where compute resources equate to financial expenditure. A security solution that demands excessive computational power can inadvertently inflate cloud workload costs, and negatively impact productivity. Organizations need to balance the need for robust security with the imperative to manage operational efficiency and cost.

Conclusion

In conclusion, the path to selecting a cybersecurity solution is complex, requiring a careful balance of independent testing insights, organizational needs, and strategic considerations. By prioritizing unbiased evaluations, tailoring choices to specific organizational contexts, and focusing on comprehensive, efficient threat mitigation, companies can fortify their defenses against the ever-evolving threat landscape. The journey toward cybersecurity resilience is ongoing, but with the right approach, organizations can navigate this challenging terrain with confidence and clarity.