60 Hurts per Second – How We Got Access to Enough Solar Power to Run the United States

The electricity grid – the buzzing, crackling marvel that supplies the lifeblood of modernity - is by far the largest structure humanity ever built. It’s so big, in fact, that few people even notice it, like a fish can’t see the ocean.

Until the grid goes down, that is. Then, like the fish dangling from the angler’s hook, we see our vulnerability. Modernity dissolves into a sudden silence, followed by the repeated flick of a light switch, and a howl of panic at the prospect of missed appointments, ruined dinners, lost workdays, stopped elevators and dark, cold evenings, and worse.

But most of the time, the grid hums along just fine ...

Now, take that labyrinthine system and add another complex network, this one delivering hundreds of gigawatts of power from the sun - solar power infrastructure. This ever-fluctuating source of energy is tapped via rooftop panels, distant solar farms, slapdash home arrays and other evolving infrastructure around the globe.

And with complexity comes risk ...

Then, take those two systems and intersect them with the Internet of Things, an ethereal network that runs from the limits of human thought – both positive and malicious – into the routines of daily life through seemingly innocuous gadgets.

At that intersection we find the mundane world of inverters and controllers.

And that's where a group of researchers stumbled upon an alarming fact:

a solar grid responsible for 195 gigawatts, or 20 percent of the world’s solar power output, is vulnerable. That’s enough to power the entire United States - and it’s just waiting to be hijacked.

Key Findings

• Bitdefender researchers have identified a series of vulnerabilities in PV plant management platforms operated by Solarman and Deye.
• This platform is responsible for coordinating production operations of millions of solar installations worldwide generating a whopping output of approximately 195 GW of solar power (20% of the global solar production)
• If exploited, these vulnerabilities could allow an attacker to control inverter settings that could take parts of the grid down, potentially causing blackouts.
• These vulnerabilities have been communicated to the affected vendors and fixed.

Understanding the Power Grid

The power grid is a vast network of power plants, transmission lines and distribution systems that delivers electricity from producers to consumers. It consists of three main components:

1. Generation: Power plants generate electricity using various sources, including coal, natural gas, nuclear, hydro and, increasingly, solar and wind.
2. Transmission: High-voltage transmission lines carry electricity over long distances from power plants to substations.
3. Distribution: Substations lower the voltage for safe distribution to homes and businesses through local power lines.

The grid balances supply and demand in real time, ensuring that electricity is available when needed. When generation exceeds demand, voltage in the grid rises to dangerous levels that can damage sensitive equipment connected to it.

When generation can't meet demand, anomalies begin to occur. Initially, your microwave oven may get several minutes slower (as it computes its current time based on the “ticks” in the network measured in Hertz: 60 Hertz per second in the US, and 50 Hz per second in Europe). Secondly, parts of the grid where the imbalance is detected are disconnected before they drag the whole system into a downwards spiral that can take days to recover from.

Fascinated? To learn more about how the grid works, you might want to watch this episode of Practical Engineering.

The Role of Solar Energy in the Grid

Solar energy has a powerful impact on the energy landscape. Solar panels convert sunlight into a clean, renewable source of power, which can be used immediately, stored in batteries, or fed into the grid.

Solar power systems are often decentralized, meaning they are installed on individual homes or businesses rather than centralized power plants. This means that generation and distribution often take place in the same ZIP code and (with some exceptions such as solar power production in parks), disruptions at this level have a limited geographic impact.

This decentralization presents several challenges and opportunities for the grid. Solar power reduces dependence on fossil fuels, lowers greenhouse gas emissions, and can enhance grid resilience by distributing power generation. However, integrating numerous small-scale solar systems into the grid can complicate management.

Solar setups produce variable amounts of electricity, and only during daytime.  Maintaining voltage levels and managing the variability of solar power must  be outsourced to a pre-programmed critical component called an inverter. An inverter converts the direct current (DC) electricity generated by solar panels into alternating current (AC) electricity, the standard used by most homes and businesses, as well as for transmission over the power grid.

The inverter performs two primary functions in a solar power system:

• DC to AC Conversion: Solar panels generate DC electricity  however, most electrical appliances and the power grid operate on AC electricity, which alternates in polarity and voltage. The inverter converts the DC electricity into AC electricity to  be used by household appliances or fed into the grid.
• Grid Synchronization: In grid-tied solar power systems, the inverter  synchronizes the phase and frequency of the AC output with the grid. This ensures that the electricity generated by the solar panels is compatible with the grid and can be safely exported to it. Differences in phase and voltage can crash the grid.  Power distributors and governments  see any deliberate attempts to bypass these grid safety measures as a threat to national security.

Security Flaws in Solar Management Platforms

Bitdefender researchers  started looking into potential cybercriminal use of  inverters, as they are becoming increasingly visible on the radar of internet-connected devices. By the foresight of “we had one already available and we wanted to test it,” we  looked into data loggers made by Solarman that are a popular choice for monitoring Deye inverter units..

To add context, Solarman is one of the world’s largest photovoltaic (PV) monitoring and management platform, and equipment connected to this platform is apparently responsible for the production of over 195 gigawatts of power across 2M+ active PV plants involving "10M+ devices in 190+ countries and territories”.

A quick look at the platform shows Solarman is not only an equipment builder,  they also license parts of their infrastructure to other manufacturers (see the Appendix). Our analysis shows that the Solarman API architecture has multiple entry points exposed for multiple companies selling inverters, data loggers and a wide range of PV equipment. The investigation of the Deye inverter and Solarman data logger have uncovered severe security vulnerabilities that could be exploited to disrupt or even take down the grid.

Note - a data logger in an inverter setup records and transmits data on the solar power system's performance. It enables real-time and historical monitoring, fault detection, and remote management, ensuring the system operates efficiently and reliably. By gathering metrics such as power output, voltage and current, the data logger helps optimize energy production and usage, facilitates maintenance through timely alerts, and integrates with cloud platforms for accessible remote monitoring. This management feature improves system performance, enhances reliability, and supports informed decision-making, ultimately contributing to the longevity and effectiveness of the solar power installation.

The Solarman platform is linked to a variety of equipment and brands revolving around it. Many of these customers seem to use the entire platform as a service, while others wrap API endpoints in custom implementations and keep customer databases private.

Beyond the multiple entry-points exposed by the platform (discussed above), we sought to discover how various parts of this architecture fit together. To validate our assumptions, we successfully attempted to register a Solarman data logger into the Deye platform.

As far as we are able to tell, Deye has been using the original Solarman infrastructure up until 2024, but they have customized their implementation and spun off a new datacenter to accommodate their own user base.

The Solarman platform has two types of accounts: one is regular reporting, where the logged-in user has access to production and usage data in real time, while the other is a business (contractor) account used by the authorized installation company to control devices, modify voltage or frequency and control other power grid injection settings). With these details out of the way, it’s time to dig into how we managed to gain access to all business and regular accounts in all solar production devices made by Solarman and its partners.

Vulnerabilities in the Solarman platform

Our research on the Solarman platform uncovered the following vulnerabilities: (These flaws are detailed in a separate paper available below.)

1. Full Account Takeover: The Solarman platform’s /oauth2-s/oauth/token API endpoint lets attackers generate authorization tokens for any account. By modifying the JSON payload, attackers can gain control over any regular or business account. This means that a malicious user could iterate through all accounts, take over any of them and modify inverter parameters or change the way the inverter interacts with the grid.
2. Token Reuse Across Platforms: JWT tokens issued by the Deye Cloud platform are also valid on the Solarman platform, granting full access to the accounts based on their ID. This cross-platform vulnerability can lead to widespread unauthorized access in the absence of well-defined boundaries to isolate vendors.
3. Excessive Data Exposure: The Solarman platform’s API endpoints return excessive information about organizations, including private details such as email addresses and phone numbers. This information can be harvested to exploit the platform’s users and organizations. In addition to personally identifiable information, an attacker can chain this with the first vulnerability to run subsequent queries on the account and obtain GPS coordinates for solar installations as well as their real-time production capacity.

Vulnerabilities in the Deye platform

1. Hard-Coded Credentials: The Deye platform uses a hard-coded account (`[email protected]` with password 123456) to access device data. This account can obtain an authorization token that grants access to any device, exposing sensitive information such as software versions, Wi-Fi credentials, and more​ (see the Deye disclosure report available below)​.
2. Information Leakage: The API endpoint /user-s/acc/orgs on eu1.deyecloud.com returns excessive private information about users, including names, email addresses, phone numbers and user IDs. This endpoint can be exploited to leak information about all users on the platform​.
3. Authorization Token Generation: The /oauth-s/oauth/token API endpoint has the same vulnerability as the endpoint used by Solarman. However, the JWT token that is generated contains a wrong value and is not accepted by the server, even though the token is valid and signed. This issue could pose a future risk if not properly patched​.

Implications for Grid Security

While most of our previous findings have a serious impact on the individual or on the internet itself, the flaws detailed in this research are fundamentally different. Access to devices interacting with the grid can have devastating effects on the proper functioning of the grid itself. These vulnerabilities pose a significant threat to grid security in several ways:

• Unauthorized Control: Attackers can take over accounts and control solar inverters, disrupting power generation and potentially causing voltage fluctuations.
• Data Breaches: Sensitive information about users and organizations can be leaked, leading to privacy violations, information harvesting, targeted phishing attacks or other malicious activities.
• Operational Disruptions: By accessing and modifying settings on solar inverters, attackers can cause widespread disruptions in power distribution, impacting grid stability and potentially leading to blackouts.

Disclosure Timeline

Solarman:

• May 22, 2024: Bitdefender reaches out to Solarman for a security contact
• May 23, 2024: Bitdefender gets in touch with Solarman security team and sends vulnerability information
• May 24, 2024: Vulnerabilities acknowledged; account takeover gets immediately fixed
• Jun 17, 2024: Vendor confirms that a fix for the API returning too much information is in place.
• Jul 17, 2024: Vendor confirms that the Deye token reuse issue is now fixed.

Deye:

• May 22, 2024: Bitdefender reaches out to Deye for a security contact
• Jun 03, 2024: Authorized security partner asks Bitdefender for details
• Jun 06, 2024: Bitdefender sends the vulnerability report
• Jun 17, 2024: Bitdefender asks for an update on the reported issues
• Jul 09, 2024: Deye sends an overview of the fixed issues
• Aug 07, 2024: This report becomes public as per the coordinated vulnerability disclosure protocols.

Conclusion

Integrating solar power into the grid offers immense benefits, but it also introduces attack surfaces that equipment makers must take into account. The security flaws found in the Deye and Solarman platforms highlight the need for robust cybersecurity in managing solar energy systems, as well as in general IoT setups.

Protecting the grid from cyber threats is crucial to ensuring reliable and secure power for all. As we continue to embrace renewable energy, we must also remain vigilant and proactive in securing our energy infrastructure against evolving threats.

We’d like to thank the vendors involved for acknowledging and fixing these vulnerabilities in a timely manner.

Too long for a read? Watch this piece of research as a video presentation on August 9 at 4:30PM (Creator Stage 2 @ the IoT Village talks) at Defcon. If you are attending Defcon 32 in person, come see the talk live or meet and greet the researchers.

Appendix

Here is a list of Solarman partner companies using the platform and potentially affected by the reported vulnerabilities. Our testing reveals that most of these vendors have scheduled maintentance windows to deploy security patches. We presume that these scheduled maintenance intervals are set to implement mitigations for the issues we reported to Solarman and Deye.