Technical Advisory: Immediately Patch Your VMware ESXi Servers Targeted by Opportunistic Threat Actors

Quick overview of VMware ESXi

Last week, unknown threat actors started targeting, en masse, VMware ESXi hypervisors using CVE-2021-21974, an easily exploitable pre-authorization remote code execution vulnerability. Experts from Bitdefender Labs have been monitoring these exploitation attempts. Guided by our telemetry, we are providing a technical advisory to describe these attacks and document our own detections in the wild. We have also included recommendations based on our observations.  

The vulnerability (CVSSv3 score 8.8) was first reported in March 2021 by security researcher Lucas Leong, and quickly became one of the most exploited vulnerabilities in 2021/2022. A common tactic by modern threat actors is to launch widespread opportunistic attacks on popular services like Apache, Microsoft Exchange, or VMware, as history shows there are still thousands of vulnerable servers even years after patch is released.  

Hybrid attacks take advantage of these opportunities by combining an automated initial phase with a follow-up manual phase. In this scenario, a compromised small company or contractor can be part of the supply chain for a much larger corporation. While the first target to be compromised may be modest, if that target has relationships with much larger targets, the exposure increases exponentially. For modern threat actors, it is often not what you have that’s valuable, it is whom you know and are connected to. 

Fig 1 – Hybrid attacks start with opportunistic vulnerability exploits 

ESXi vulnerability analysis

The core of this vulnerability is a service discovery protocol called the Service Location Protocol (SLP). SLP service on ESXi hosts parses network input without authentication and runs as root. The combination of widespread use, low attack complexity, no authentication requirement, and zero user interaction makes SLP service a lucrative target. 

This is a VMware custom implementation of OpenSLP, an open-source project, but with additional hardening (and unfortunately bugs). The protocol allows discovery of location and configuration of network services on a network – think of it as a virtual index page or shopping mall directory. The vulnerability exists in only the VMware implementation of SLP. Other projects based on the OpenSLP project are not vulnerable. 

OpenSLP uses port 427 (TCP/UDP) that can be exposed to the Internet. Since 2021, VMware has recommended disabling OpenSLP service, and ESXi 7.0 U2c and ESXi 8.0 disable the service by default. Our analysis shows there are still tens of thousands of vulnerable VMware ESXi hosts exposed to the Internet. 

Figure 2 – Versions of publicly exposed VMware ESXi servers. Source: Shodan 

 

One of the challenges related to SLP is discovering vulnerable systems. When scanning for vulnerable VMware ESXi servers with SLP enabled, the returned results are not conclusive. This is because SLP does not return any information about running service (banner) without receiving specific input first. Scanners like Shodan or Censys rely on banners to identify running services. The best discovery method is to use port scanners with an input query built to trigger a response from SLP. 

The fastest mitigation for Internet-exposed hosts is to block access to port 427. While this mitigation will slow down threat actors, it should only be considered as a temporary solution until the vulnerable servers are updated – if you need a quick workaround, we recommend disabling SLP. While public conversation is focused on Internet-exposed OpenSLP ports, there are other ways to exploit this vulnerability. The SLP service is available to all virtual machines running on a vulnerable ESXi host. By compromising one of these virtual machines, threat actors can escape from a compromised virtual machine and take over the hypervisor to gain access to other virtual machines running on the same server.  

As we’ve observed in our technical advisory on Proxy*Hell exploits, many companies rely on mitigation workarounds instead of patching or upgrading vulnerable legacy systems. This is beneficial for threat actors. If the underlying vulnerability is still present, they just need to find a method to bypass the mitigation. 

Figure 3 – Even if port 427 is blocked, VMware ESXi host remains vulnerable 

 

As for the actual payload that we are detecting, it’s mostly Trojan.Ransom.ESXiArgs.*  variants. ESXiArgs ransomware is targeting virtual machine files with extensions *.vmdk, *.vmx, *.vmxf, *.vmsd, *.vmsn, *.vswp, *.vmss, *.nvram, and *.vmem. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has released a script to recover files encrypted by ESXiArgs ransomware. 

It should be noted that the final payload can vary. While ransomware payloads like ESXiArgs are easy to detect (after all, ransomware attacks are designed to be disruptive), other payloads can be stealthier. Initial access brokers can deploy a remote webshell, disable SLP service to prevent other threat actors from exploiting the same vulnerability, and wait for opportunity to monetize the compromised network. We recommend conducting a threat hunting exercise to identify potential hidden threat actors. 

Ransomware groups started adopting more exotic programing languages like Go or Rust. One of the advantages of using these languages is that threat actors can target different operating systems - including hypervisors and Linux-based virtual machines. The current wave of ESXiArgs attacks is not a polished operation, and we can expect more experienced groups to start targeting other operating systems this year. 

Conclusion & recommendations

The vulnerability in VMware ESXi is a clear reminder of the importance of keeping systems up to date with the latest security patches while also employing strong perimeter defense. Attackers don't need to scour for new exploits or novel techniques when they know that many organizations are vulnerable to older exploits due, in part, to the lack of proper patch management and risk management. 

In addition to prevention and cyber hygiene, multi-layered protection on all endpoints, servers, and workloads is critical. In our telemetry, we have identified the following indicators of compromise detected by different endpoint security modules:  

 

SHA256 Hash	File type
11b1b2375d9d840912cfd1f0d0d04d93ed0cddb0ae4ddb550a5b62cd044d6b66	ELF
773d147a031d8ef06ee8ec20b614a4fd9733668efeb2b05aa03e36baaf082878	Python script (.py)
10c3b6b03a9bf105d264a8e7f30dcab0a6c59a414529b0af0a6bd9f1d2984459	Shell script (.sh)
5a9448964178a7ad3e8ac509c06762e418280c864c1d3c2c4230422df2c66722	Shell script (.sh)
87961344f13a452fb4aa46dd22a9aa31c5d411b1d8d37bac7a36f94a5be9fb0d	Shell script (.sh)

 

Implementing IP, domain, and URL reputation is one of the most effective methods of defeating automated vulnerability exploits. According to analysis in the Data Breach Investigations Report 2022, only 0.4% of the IPs that attempted Remote Code Execution were not seen in a previous attack. Blocking bad IPs, domains, or URLs on all devices, including remote and work-from-home endpoints, can be highly effective. 

Finally, companies of all sizes should implement detection and response capabilities to detect any suspicious activity on the network and minimize the dwell time of adversaries. The GravityZone XDR sensors detect suspicious activity on the server or virtual machine, and alert security teams to lateral movement attempts or the establishment of an external connection by the threat actor. This technology can be augmented by good security operations, either in-house or through a managed service like Bitdefender MDR. 

Another key aspect of the attack is that the attackers are targeting configuration files on the ESXi hosts. A system like GravityZone Integrity Monitoring identifies changes made to important files. This is done by assessing differences against a baseline known-good state of the monitored entities. Having such a solution in place will help thwart the ability of the attackers to compromise the systems without being detected.  

Ultimately, it is important to keep in mind the best protection against modern cyber-attacks is a multi-layered defense-in-depth architecture. Bitdefender GravityZone covers this by delivering prevention, protection, and detection and response in a single solution. 

Learn more about how Bitdefender GravityZone protects your organization.