Interview: Brian Chess, SVP of Infrastructure & Security at NetSuite

Jeff Williams: You were the inventor of one of the most popular application security technologies out there, and I'm wondering how you reconcile that with this belief in very early training of developers and being immersed in security. Can we do it with technology, or does it have to be much more?

Brian Chess: I think your point is that by applying a static analysis tool you can find some vulnerabilities in software, but in order for there to be the vulnerabilities in software somebody has to already have made a mistake, and wouldn't you be better off if you just didn't make a mistake in the first place?

Jeff Williams: My question was really, how do we get developers doing this? Do we give them assistive technology that can allow ordinary developers with no real training in security to write secure code, or do we have to spend a lot of time training them? I've trained thousands of developers personally, so I'm deeply invested in this approach.

Brian Chess: My favorite analogy here is riding a bike. When you learn to ride a bike you make extreme errors in balance. Usually that results in some skinned knees. It certainly results in some major disruptions to your bike ride because you're on the ground, you've got to pick yourself up and try it again, and it's scary.

Over time you don't actually make any fewer errors. When you're balancing on a bicycle you're constantly making small errors, but you've learned to correct them very, very quickly before they affect your bike ride at all to the point where you don't think of them as errors. It's what it means to ride your bike. It's how you balance the bicycle.

I think that eventually that's where we would go with security and programming. When I write software, I make errors all the time. These days most of them get caught before you even try and compile. Then, another chunk of errors gets caught when you do compile, and another chunk of errors gets caught as soon as your program tries to execute. Then, a relatively few number of errors actually make it through to when you try to do something meaningful with your program. I think that security is going to take the same path as we get better and better about teaching people to balance.

Jeff Williams: Balance obviously requires instant feedback. How critical do you think that is as part of the development process?

Brian Chess: I think the very first feedback starts in your head, which is why it's so upsetting to me when we teach a kid that the right thing to do is just run all their code as root. Because that is the very beginning of starting to install the wrong intuition. The tighter the feedback loop can be or if the mistake never leaves your brain that's ideal. If it makes it from your brain onto the computer, but we tell you about it instantly, that's a little less ideal, but it's still really good. The tighter the feedback loop can be, the better.

Jeff Williams: Yeah, I really like thinking about that feedback loop and how we can shorten it. Let me ask you this. Speaking of shortened feedback loops, agile software development and dev ops have been advancing incredibly rapidly. I'm surprised how fast it's permeated a lot of the large financial enterprises that I do work with. It seems to have some challenges for the traditional way we do application security. What do you think of agile and dev ops, and how we evolve to make those processes more secure?

Brian Chess: Well, I'm a little leery of this area just because I think even though agile has been around for a while now, dev ops is younger I would say, but for both of these terms they get interpreted a number of different ways. If you really boil them down to the least common denominator there are technology people in these companies saying I want to do things differently, and I think I can do them more efficiently than the generation that preceded me.

For the most part, they're right. If I think about dev ops in particular, the startup companies that I've worked at always have dev ops when they first begin.

Actually, when I first started at NetSuite in 1999 we had dev ops because there were no ops. There were only developers, so very necessarily it was dev ops. We were looking at the only server because it was sitting on a milk crate in the middle of the room. Then, I think we advanced to where there was somebody who had more system administration skills. We thought we advanced, anyway, to somebody who had more system administration skills that we would sort of chuck things over to if we wanted to deploy them.

Now we're kind of moving backwards and saying no, we're going to automate everything. That means that the operations position is really just a developer working on a different set of features. That allows us to iterate much more rapidly, and it screws up the security processes that expected there to be a gate.

Jeff Williams: Right.

Brian Chess: Because now the gate is much, much harder to establish and also runs counter to this idea that we're going to be able to move very, very rapidly. Because the guys are trying to push the code out the door while the security person is still saying you want to do what exactly, what is this going to achieve. Too late.

I don't see how in the long term you can be successful with merging those roles if you don't merge the security role along with them. That, I don't think, is so widely acknowledged quite yet.

Jeff Williams: I think that's fair. Actually, Nick Galbreath was on a few weeks ago, and we were talking about this. He made an interesting observation. He said actually dev ops represents an interesting opportunity for application security because it forces organizations that might not have a lot of development process and specifically technology infrastructure to put that stuff in place. The same infrastructure that you use for automated testing and continuous integration and continuous delivery provides a huge opportunity for security if we're smart enough to build the right things into that process. It might be a huge step up for apps dev.

Brian Chess: I think the potential is there, but I can't say that we've entirely cracked the code on how to unlock it.

One thing you can say about dev ops is we're going to radically reduce the "people cost" of getting code into production. We're just going to automate a ton more stuff. On the security side, I think we haven't necessarily followed suit. We haven't said we know how to take the people out of the equation to the same degree that they have figured out how to do it when they're talking about deploying code.

Jeff Williams: I think that's an interesting way of talking about it. How do we get the people out? Because, frankly, we don't have enough people to do security work, so we've got to figure out a way to get them out or there will always be a bottleneck.

Brian Chess: That's true. We don't have enough of them, and they're pricy. They're pricy in terms of time and money.