Interview: Gene Kim, Former CTO of Tripwire

In this interview, we have the pleasure of talking with Gene Kim. Gene is a multiple award-winning CTO, researcher and author. He was founder and CTO of Tripwire for 13 years. He has written three books, including "The Visible Ops Handbook" and "The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win."

Gene is a huge fan of IT operations, and how it can enable developers to maximize throughput of features from "code complete" to "in production," without causing chaos and disruption to the IT environment.He has worked with some of the top Internet companies on improving deployment flow and increasing the rigor around IT operational processes. 

In this episode, Gene and I discuss DevOps, continuous security, and how a company develops a great security culture. 

Gene will be hosting the DevOps Enterprise Summit this fall on October 21-23, with some of the best practitioners telling their story about transformation in large, complex organizations, including GE, Macy's, US Dept of Homeland Security, Disney, etc. The complete agenda is here. Also, be sure to check out these additional resources that were referenced in the podcast:

1.  The amazing Twitter infosec talk from AppSecUSA 2012 by Justin Collins (@presidentbeef), Neil Matatall (@ndm), and Alex Smolen (@alsmola)

2.  The 2014 DevOps Survey Of Practice (Click here for the SlideShare link or the full report can be found here.)

The following is a brief excerpt of our interview.

Jeff Williams: So one of the things that I think DevOps has been revolutionary with is this idea of continuous. And it's sort of the whole point of this series of podcast is to talk about continuous. And we talk about continuous integration, continuous delivery, continuous deployment. If I said the words continuous security, what would that mean to you?

Gene Kim: Yeah, it brings a big smile to my face is what it brings. I think, from a security context, one of my favorite saying is, “Information security is a team sport.” Right? Information security, in our silo alone, cannot win the battle. We must work with our teammates, whether we're a product owner, dev, test, or ops, we must work with them in order to achieve the goal. And so, when this talks about things like continuous security, I think one reaction is like that's absolutely right on. Just like in manufacturing we know that continuous is always better than periodic. In manufacturing there's a theoretical ideal called single piece flow that means we have a batch size of one, we never have inventory, it's a pure pull through system. But without something like DevOps, it's not amenable to integrating security objectives into it.

So what continuous integration, continuous deployment, continuous delivery, what they bring is a work pattern where dev and ops are constantly checking things into production. Every time we do, it gets built on a build server, it runs a set of automated tests on test servers. Even the integration tests are run in an automated fashion. And then, ideally, deployed automatically into production. And so when those conditions exist, it's a perfect place for information security, it's a perfect conditions because that will just integrate our automated tests suites into the deployment pipeline. And so, I think when continuous security is great, but without something like continuous delivery and DevOps practices, that can’t… continuous security just can't survive. They go hand in hand.

Jeff Williams: Let me press on that a little bit. So, I love the manufacturing analogies and I know that your work is based on a lot of the different manufacturing revolutions over the years. I want to see if we can transform security into that manufacturing process, except what's missing for me is the thing that we're building. Like what is it actually that security is producing? And we had an argument about this over margaritas in California and I thought it was really a useful, interesting discussion.

So my suggestion was that security actually is building something. It's building proof that this software system is trustworthy enough to go operate. And I think if we treat security as building something, then all the DevOps, methodologies, all the tools, everything starts to fall into place for me and it makes sense. Like, “Oh, we're building assurance.” And I want to really try to drive that home, but I guess I'm wondering what's your take on that? Is security something you build or is it just something that's sort of magically mixed into the output of a manufacturing process?

Gene Kim: You know, I've thought a lot about that conversation we had in Santa Monica. And it did remind me, many weeks afterwards, I’m like, "Here's this thing I should've mentioned to Jeff." And I think it's this kind of pithy two liner. One is, the first line is security testing can be automated but true security assurance cannot be. You know, this is a slight adaptation of around QA, right? Testing can be automated but true quality can't be. And I think the real point of this is that the only way that we can free up enough cycles to have security assurance is to automate all the things that can be. And so, we as human beings, our talent shouldn't be wasted on repetitive, manual, error prone security testing. That has to be automated so that it can integrate into the continuous delivery pipeline. Then, and only then, do we free up enough fore brain power to actually do the things that only humans can do. To make the call, have we achieved a security assurance objective or not?

Jeff Williams: Right. Yeah, I couldn't agree more. There's a lot of things in the security architecture areas and threat modeling and some risk management decision making kinds of processes. Even at the executive level, strategic decisions that, yeah, I wouldn't even want to think about trying to automate. There’s so much… you know, I feel like it's this iceberg where there's 90 percent of the work is automatable if we really set our minds to it. I want to free up the limited security work power that we have available in the world. I want them focused on the top ten percent of the iceberg and not spending their time just chasing the next XSS hole or something.

Gene Kim: Right. You know, and what I love about DevOps, is that… Josh Corman in his very talent, said in a very pithy way the DevOps community is waiting for information security practitioners with open arms. And I found that so much to be true. And I think what becomes evident is that the security objectives become just like every nonfunctional requirement. Whether it's deployability, or testability, or scalability, and I think that's exactly what we want. We don't want a unique status among everyone else in the value stream or in the quark chart. What we want is that the things that need to be fixed in security should be viewed alongside the features and the other nonfunctional requirements and we, as people who want to do what's right for the organization, will do the right thing, and security will be a beneficiary of that.

Jeff Williams: So how do companies get that great security culture? What's the secret? I mean, I'd love to say, "Well, oh, you just buy this tool or you just train your developers," or whatever it is. But I've noticed some companies, actually everyone there cares about security, they it very seriously and its part of their process. Other companies that seem to be doing the same stuff, they just don’t have the cultures for it, it doesn't work, they keep making the same mistakes over and over again.

Gene Kim: You know, one of the best, easiest ways to spot, I think, a great culture that will integrate and embrace security versus one that will reject security like the immune system in whatever shape or form that we cloak it in is, is there a culture that says were going to dedicate 20 percent of all our dev and ops cycles for the nonfunctional requirements? And so, you see this absolutely in organizations like the Unicorns, the Amazons, Etsys, Google, Netflix, and so forth because they know that if you don't retire technical debt, if you don't take some of those cycles off the table and spend them, not on features, but you use them to fix problematic areas of the code, then technical debt will inevitably and invariably suck up all the air in the room and leave no time left over for everything beside the feature. In fact, technical debt will balloon so that you're spending no time on features. And that's what happened at Ebay, at Linkedin, and so forth.

So the countermeasure is that value system that says the nonfunctional requirements are different. And when those conditions exist, security becomes just another beneficiary, another facet of the quality jewel. And I think there’s another, as we talk about culture, I love this quote from Dr. Stephen Spear, he and Mike Rother are credited for helping document and decode the Toyota production system. He said, "Culture is often viewed as this airy-fairy nebulous thing that is impossible to change." And he offered a different definition. “Culture is the collective response to how a group responds to stimulus.” And, so, I think when we talk about cultures, how do we get organizations to see the value of the nonfunctional requirements, take the time to fix problematic areas, to improve flow, increase reliability, increase security, reduce the surface area of risk?

You know, those are all the things that the organization has to value and they have to react the right way. Hey, I got a gem for you, Jeff. We just wrapped up the second year of DevOps survey of practice. And so we've benchmarked 14,000 organizations. I worked with a gentleman named Jess Humboldt over two years and…

Jeff Williams: This is sort of a good to great style study, right?

Gene Kim: Oh, exactly right. It's a cross population studies. And instead of 40 organizations, which is what Jeff Collins did, we did 14,000 organizations. And, so, we found that high performers, they deploy more frequently, they can do it more quickly, they have higher change success rates, they have shorter mean time to repair, better security. One of the top predictors of performance was how organizations answered a set of questions. And the set of questions uncovered whether the organization had a culture of fear, where we shot messengers who told bad news, we discouraged bridging between functions, we’re afraid to suggest novel ideas. Versus what they call a generative culture, where we train messengers to tell bad news, failure causes genuine inquiry, we embrace novel ideas because we know that is what's required to fix complex problems. And how organizations answer that question, you know, they call it pathological, bureaucratic, or generative, was one of the top predictors of whether you were a high, medium or low-performer.

Jeff Williams: Interesting.

Gene Kim: Again, and so when you have high-trust management cultures, it not only predicted IT performance, it also predicted organizational performance. We found that the high performers were 2.5% more likely to exceed profitability, market share and productivity goals. We found that the high performers, where if you look at market cap growth over three years, they outperformed the low performers by 50%. And it also created the conditions where people felt there was a win-win relationship between dev, operations and information security. There was also a higher job satisfaction. So I think we're really zeroing in on what we need out of our culture to enable great IT performance as well as great organizational performance.