Comparing the OWASP Top 10 to the Real-World Top 10

Enterprise software is drowning in vulnerabilities and even organizations highly motivated to fix security flaws in their applications struggle to do it in a timely fashion.

Researchers in the application security world believe that the only way that organizations can truly make a dent in the risk exposure of applications is by changing the way that DevSecOps teams prioritize their fix backlog. And this means redefining what 'high-priority' is for vulnerabilities by focusing more on adversary behavior. 

How Much Flaw Volume Are We Talking?

To get an idea of the scope of vulnerability backlog faced by developers and security teams, consider some illuminating statistics from the recently released State of Software Security (SOSS) report from CA Veracode: 

• Over 85% of applications contain at least one vulnerability when they're first tested
• More than half of flaws stick around three months after they've been discovered
• One in four flaws linger more than a year after discovery 

Even vulnerabilities ranked as high and very high severity persist for extended periods of time—this year's SOSS showed that 25% of these serious flaws were not addressed within 290 days of discovery.

These extended windows of flaw persistence are particularly troubling considering how quickly attackers take advantage of new flaws when they find them. It only takes 22 days for the bad guys to develop a working exploit when they discover a vulnerability. That's a pretty lopsided race between defenders and attackers—and that's assuming that the cybercrooks are starting the race at the same time as the appsec team.

Given the proliferation of zero-day exploits online, it's safe to say that the attackers usually have a head-start in that regard.

Clearly, developers and security teams are struggling to keep up with the volume of flaw fixes in their backlogs. It's a colossal problem that can't be fixed all at once. The incrementalism of DevOps methods are helping organizations chip away at the problem more quickly with smaller, more frequent code fixes. SOSS results hinted that DevSecOps organizations that scan their applications more frequently fix flaws as much as 11.5x faster than the typical organization.

But even when flaws are fixed more quickly, there's always more of them to address. So there's an important question to be asked: are the flaws devs are asked to fix first really the ones most likely to reduce chances of attack once they're closed?

"We have to ask the question, 'Is what we're testing driving us toward finding the right issues?" says James Wickett, researcher with the firm Signal Sciences.

Refocusing Fix Priorities on Attack Behavior

Pundits like Wickett and Shannon Lietz, who works as the leader and director of DevSecOps for Intuit, say the answer is "No" if the team only prioritizes using traditional benchmarks like flaw severity or the OWASP Top 10.

The pair recently exhorted the crowd at DevOps Enterprise Summit to stop directing developer action based on "the big bullet list of what we've been told to fix" in the past and instead start using telemetry from what adversaries are actually doing in the real world to decide what to fix first. The problem is that too often security teams overwhelm developers asking for work that doesn't directly relate to what the bad guys focus on, Lietz says. This is symptomatic of a larger problem in the security world.

"When you talk about adversaries, you talk about means, motives, and opportunities. In the security industry we focus a lot on opportunities—if we block out the opportunity then the bad guys are going to go away. But the truth is that we're not really driving out those bad guys. We're just seeing them change how they think about opportunities."

This thought process was the basis for them developing what they call the "Real World Top 10" as opposed to the OWASP Top 10. The idea is to reconcile the list of most common and serious coding mistakes made by developers against the list of the most common vulnerabilities attackers are using to target an organization's applications.

"Vulnerabilities on the OWASP Top 10 are the most recognized part of the problem, but it doesn't reflect attacks in the real world. It's only the tip of the iceberg," Wicket explains. "The security industry leads us to believe that we can focus on this stuff and that's going to solve our problem. But there's other pieces to it."

According to real-world attack statistics collected by his firm, only about 5% to 10% of attacks go after vulnerabilities on the OWASP Top 10. He and Lietz took that information and combined it with other attack data from honeypots, white hat security researcher data and other telemetry that added up to two petabytes of data with which they could model attack behavior to come up with their Real-World Top 10. Compared head-to-head with OWASP Top 10, it looked like this: 

The point here is that the list on the right isn't universal—it'll change based on the data collected by each organization with regard to how attackers are interacting with their specific applications. Instead, it's important to look at it and notice how different such a real-world is from the risks delineated by OWASP.

"With the Real-World Top 10, you're instrumenting your application looking for things like anomalies to make it so you have less guessing," Lietz explains. It's essentially answering that long-asked question of how can a developer know the time they spend on security is actually going to make software safer.

The Takeaway For Devs

That's not to say that Lietz and Wickett didn't have some universal advice for DevSecOps teams when it comes to prioritization. Once they looked at the scope of their research they did see major themes bubbling up that would be useful to communicate to developer communities.

Among those was what they call the AppSec Hierarchy of Needs. Fashioned after Maslow's Hierarchy, this is a pyramid of five specific areas of security that could solve many of the biggest application security problems out there today. It's basically a list of five principles to live by

"If I can give a developer five things to keep track of to use against the bad guys, I'd say you've got to zone and contain your application decently. You've got to be able to have asset management, logging, authentication, and encryption," she explains. "If you get 80 percent of the things there, you'll make your workflow safer."