The State of Endpoint Security in Virtual Environments

There’s no question that the majority of organizations are virtualizing servers, and increasingly, desktops within their environments. With this shift comes a plethora of new risks. We’re getting better at porting network security platforms to a virtual format, primarily firewalls and intrusion detection and prevention systems. Encryption for virtual and cloud environments is also slowly improving. Another area that seems to be evolving is endpoint security.

In some ways, the challenges of endpoint security are more complex than some others, for a few different reasons. First, endpoint security has to scale across a larger number of systems, in many cases. In addition, traditional endpoint security products are usually agent-based, and consume significant amounts of resources (disk, memory, and CPU). This can easily throttle a shared infrastructure environment.

There are a number of different architectural and implementation models for virtual endpoint security today. Each has benefits and drawbacks, which I’ll touch on. For many organizations, the most common tools implemented to protect virtual machines are signature-based endpoint security solutions like antivirus.

With traditional antivirus and endpoint security solutions, agents on individual VMs execute malware detection scans regularly (often real-time scans for certain actions, as well), and these agents need to be updated with the newest signature definition files very often (daily is common).

The advantage of using this kind of tool is primarily familiarity and comfort with the installation and monitoring, much of which may already be in place within the security and IT operations teams.

However, using traditional antivirus and endpoint security agents is not a good solution in many ways. Most definition files have become large and unwieldy, which can take up significant amounts of storage and lead to somewhat complicated update processes. In addition, deployment of new VMs may even be hindered by the size of templates and VMs with large definition files installed. Scans can take a long time and will likely require significant resources drawn from a central pool available to the entire virtualization environment. If antivirus agents poll a central server or online vendor site for updates simultaneously, a denial-of-service within the virtual environment could ensue!

The second model commonly used for endpoint security deployment within virtual environments today is the “agentless” model. With this type of solution, a dedicated virtual appliance (a specialized VM) is installed on the same hypervisor as the VMs to protect, and hypervisor APIs and VM drivers are used to integrate with VMs communicating with the underlying hypervisor kernel to facilitate “offloading” protection of the VM, its communications, and its operating system and disk. VMware vShield Endpoint is a good example of a vendor-specific offering that facilitates this approach.

The biggest advantage this offers is performance improvements, as well as efficiency due to central signature updates for the virtual appliance instead of endpoint updates. The downsides are potentially significant, though. First, these types of solutions may not have the level of system-level visibility as installed agents would. More advanced threats today are memory-resident only, so there are no malicious files or changed components within the OS installation.

In some cases, malware is highly application-specific and may only affect a very small subset of memory dedicated to one or more applications (like a browser or messaging client). Many agentless solutions don’t have the ability to properly monitor and detect threats that are only in memory. Most are also signature-based, too, meaning there is little to no behavioral monitoring that may indicate new or unusual exploits or malware not detected by signatures.

A new breed of endpoint tools is emerging that blends both approaches. Some tools install a lightweight agent that also communicates directly with a virtual appliance to minimize the footprint on the VM, while still providing some local access to the VM’s components that may be prime targets for compromise.

Some of the other considerations that endpoint security must accommodate include the following:

√ Mature enterprises are often unwilling to replace a “tried-and-true” solution with a new vendor product that is not as well known. Newer products may need to partner with leading hypervisor vendors to be accepted and integrated into the security controls landscape.

√ Hybrid and public cloud deployment architectures can pose a real challenge to endpoint security products that are able to function only with deep integration to the hypervisor. In these cases, public cloud providers may not support the extension of their hypervisor kernels or integration with native hypervisor APIs, reducing or even eliminating the possibility of using entirely agentless approaches.

√ In some ways, deeper integration to the hypervisor may be needed to truly allow thorough introspection to the various virtual machine hardware files and components in real-time. When antivirus and other endpoint security options are still operating at the application layer in a virtual environment, more overhead will always be required. To truly provide maximum monitoring and detection capabilities with minimal consumption of resources, a very deep integration into the hypervisor kernel may be needed.

In the world of virtualization and cloud security, another key point to consider is the gradual shift toward including the CPU, or physical chipset, in the conversation. With Intel’s VT-x technology, and many vendors providing integrity and cryptographic support and validation through chipset features, the way that endpoint security is implemented may also trend in this direction over time.

Should the focus be more on hardware integration or deeper hooks into the hypervisor?

Do agent-based, agentless, or hybrid antimalware and endpoint security tools make the most sense in high-density virtual environments?

What solutions will provide the most seamless architectural and operational shift from on-premise installation to cloud service provider infrastructure, with no loss of introspection and monitoring?

These are all open questions at the moment, and the pace of technology evolution in the endpoint security space is moving very quickly.

There are a lot of promising new tools and technologies emerging, and the hypervisor vendors are rapidly changing at the same time, which makes this a difficult challenge, indeed.

I’ll be writing about more virtualization and security topics here in the coming weeks and months, so check back with us!