7 Habits of Highly Successful (And Secure) Rugged DevOps Teams

The DevOps discussions kicked off with a full slate of speakers at the DevOps Connect: Rugged DevOps event at the beginning of the week. One of the highlights of the day was a talk by Amy DeMartine, senior analyst for Forrester, who offered up seven of the most common practices among forward-looking IT teams who have managed to implement DevOps while maintaining and even improving security standards--essentially establishing Rugged DevOps practices. 

The idea behind these habits is enabling development and ops teams to move faster by streamlining security and more smoothly inserting it into the continuous delivery workflow. Here are some of the highlights from DeMartine's suggestions.

1. Increase trust and transparency between Dev, Sec and Ops

Before this first habit is established, all others will remain an uphill battle. DeMartine says that traditionally structured IT departments struggle in the shift to DevOps due to the stereotypes that developers, operations people and security staff have about one another. Developers are seen as the cowboys who do whatever they want. Operations are seen as the department of "No." And security is locked into the role of the persistent nagger.

Successful Rugged DevOps teams break out of that stereotyping through exercises in empathy, which in turn build trust. This starts by understanding what the other groups of people are challenged with in their daily lives and speaking to those concerns when communicating with them. So, security and operations staff need to understand the bane of existence for developers is unplanned, unscheduled work. Meanwhile developers and security must keep in mind that infrastructure and operations staff are constantly battling downtime and performance glitches. And developers and operations folk must recognize that breaches and vulnerabilities are the number one problem security folks face.

"So a part of it is, when you're reaching out to these people, use the right language. If you're talking to an operations person, talk about outages and performance glitches. If you're talking to a developer, say, “I can reduce your unplanned, unscheduled work.” So if we use common language to understand we're all on the same team, that’s one step forward—simple step, but important."

2. Understand the probability and impact of specific risks

Security folk that want DevOps teams to keep infosec concerns top-of-mind need to find a better way to get them to break out of the tunnel vision of their everyday work to understand how risk probability and impacts will affect the software that is marching its way to production. 

This means security pros need to get better at bringing risk to conversations and increasing the knowledge and visibility around the risks specific to the business.

"You have to make it real for developers and for operations people—which can kind of be a trick. My suggestion is to use real life examples and discuss. Pull up the latest, headline grabbing breach, and start talking about it," DeMartine says. "Could this happen to us? What do we have in place that makes it so that we are protected against it? What would we do if we found it?"

3. Discard detailed security road maps in favor of incremental improvements

Just as developers and operations staff have had to break away from behemoth waterfall projects, security teams need to shift away from vastly complicated and detailed security road maps if they're going to make a meaningful difference in the DevOps world, DeMartine said. This means instead establishing a vision and seeking out areas for incremental improvements.

"Now, this is a mind shift change that has to happen—developers do that change with Agile. They break out things into smaller and smaller pieces. For infrastructure and operations, it freaks us out, but we too have to get in the mindset of releasing smaller changes faster," she said. "For security it is a matter of letting go of detailed security road maps."

4. Use the continuous delivery pipeline to incrementally improve security practices

One of the foundations of DevOps is the continuous delivery pipeline and its attendant automated tool chain. This linked set of tools for development, integration, testing, deployment and monitoring of code throughout the lifecycle is the lifeblood of an efficient continuous integration/continuous delivery process. And DeMartine says that security has a real opportunity to take advantage of the pipeline to insert security tools in a way that can steadily improve security metrics.

"So what do you insert in this continuous delivery pipeline? Pretty much everything. Now, the trick here is, these tools take time, and if your team is delivering every hour, they will not appreciate a test that takes a day," she explained. "So you've got to be careful about what you suggest to insert in here and how fast those tools can run. But certainly, pretty much all of those tools can fit in."

This is where the security team can add value, by helping to plan where and how to insert testing and security gating throughout the process without tripping up the fast pace of DevOps software delivery.

5. Standardize third-party software and then keep current

Heartbleed, Shellshock and now DROWN. The past several years have been awash in headline-splashing examples of how third-party components in everyday enterprise software and web assets puts everyone at risk.

DevOps has only accelerated the dependency of IT shops on third-party software components as Agile development teams seek ways to efficiently develop software without reinventing the wheel. But in order for shops to truly ruggedize and secure DevOps patterns, there needs to be some method to the madness, warns DeMartine.

"And I know this seems really obvious, but the best way and easiest way to make that habit is to create a component library/ If you make it easy, the developers will come. It’s that simple—and even better, make it invisible—and they will come," she said. "You want to reduce the number of versions of anything."

6. Govern with automated audit trails

Contrary to many security folks' kneejerk fears, DevOps is not necessarily the wild west. There are ways to institute separation of duties and keep track of who touches what--it is just a matter of harnessing the power of the automated systems put in place within the continuous delivery pipeline.

Even though in many cases the old methods of instituting security approvals has gone by the wayside, the advantage of these tools is that they're all generating audit trails, DeMartine explains.

She says that security teams should be following a multi-step process to govern with automated audit trails. The first is to create automated security alerts within the production enviornment to understand when intrusions happen. The second is to work their way back to the requirements stage to prioritize the most sensitive systems for security approvals--things like authorization or authentication systems, for example.  Everything else is left to the system to automatically push code through, so that "only the tools themselves are allocating hardware and software. Then it is up to security to use the audit trail this offers to look for drift away from the norm.

This kind of drift will offer clues into whether outsiders or insiders were touching systems that they weren't supposed to.

7. Test preparedness with security games

The seventh habit is the one that DeMartine calls the "most fun" but the least likely to be practiced. That is finding a way to fold in red team/blue team exercises into the software development lifecycle.

The key is getting developers, operations and security staff all involved and rotating members so that everyone gets to participate on both red team and blue team duties. She says it can happen regularly or intermittently, but the idea is to use findings to make changes to tool sets and fixes to daily practices.

"When you do this red teaming, you're really going to focus on metrics. How fast were you able to detect the problem? How fast were you at remediating? How fast were the developers in pushing fixes through the continuous delivery pipeline into production. And is this an attack that has been tested for or do we need to augment our testing so it doesn't happen again?"