Strategic evolution of DevSecOps: Interview with Larry Maccherone at RSA Conference 2024

This is TechStrong TV.

Alan:

Hey, everyone. We're back here live at RSA in San Francisco. We are our broadcast alley, which happens to be in Moscone West where a lot of keynotes take place, not the expo hall and so forth.

But, you know, we are doing something a little different this year. We're filming out into the crowd to give you guys a sense of what's going on behind us. We could turn around and you'd see the texture of logos, but we think this is more interesting.

Speaking of interesting, let me introduce you to our next guest. He's a friend of mine. For people in the DevSecOps world, he doesn't really need a big introduction, but maybe some of you don't know him. And I'm gonna try to pronounce it right because I try to give it that Italian flair. My friend, Larry Maccherone.

Larry:

Very good. Very good. Yeah. Grandma would say it was wrong, but Grandma would say when I say it, it's wrong too.

Alan:

Yeah. I'm only listening to her. She's making sauce or something. 

Larry:

Yeah. You're right. Right. Right.

Alan:

Well, God bless her. Right? Rest in peace. Anyway, Larry, if you don't know, is with Contrast Security. Before that though, when I first met Larry 10 years ago, eight years ago, DevSecOps was not yet a thing.

And we started doing this DevSecOps event here at RSA, trying to bring together the security tribe and the developer community to work together. Larry was one of the few people I met who actually got it and was doing it. You were at a large … well, it was Comcast. We're allowed to say that.

Larry:

Yeah. Yeah. Of course. That's my background. I was the head of application security at Comcast. And, just so people understand, most people over here think of Comcast as that cable company. But cable is a small sliver of their business. I mean, NBC, Universal, Sky, DreamWorks, Peloton, Hulu, they all had a piece of tech. I mean, the global media conglomerate, but ten thousand developers. Not only developers, how many different apps? We didn't know. We never knew. That's crazy. We had headcount because we had to pay those people. And we knew their job categories, so we knew how many developers we had to support.

Alan:

Larry, you know, as I said, you were one of the first people I met that really got that whole DevSecOps thing about how AppSec and AppSec testing needed to be done with developers.

Not necessarily having the developers do it. Developers have enough to do, certainly. But AppSec testing needed to be done to the left, shifting left of the deployment cycle.

And you spoke at a bunch of our events here and at other events. And then — what was it, about four years ago now? — you went to Contrast. Three years ago?

Larry:

Three years ago. Yeah. Yeah.

Alan:

Let's talk a little bit about that phase of your career here. Now three years ago, you moved to Contrast. And for people who don't know, Contrast is one of the leading application security providers based out of Maryland.

Larry:

Well, it was founded out of Maryland, and our founder still lives there, Jeff Williams. But our headquarters is officially here in the Valley.

Alan:

Really? Oh, I didn't know that.

Larry:

Yeah. It has been for a few years. So yeah.

Alan:

Very cool. I didn't realize. And then, well, I don't wanna tell the Contrast story. You tell the Contrast story.

Larry:

Well, I can stitch the Comcast and the Contrast story together. So when I was at Comcast as the head of application security, we had, you know, 600 different development teams, a lot of diversity there. I had to support all of them, and there was no way to throw it over the wall; gatekeeping, policing, policy enforcement approach was never gonna keep up with the rapidly accelerating pace of development. So I started on this sort of shift left, DevSecOps approach to doing it.

And prior to that, years before that, I had launched this thing called Build Security In with Gary McGraw and Noopur Davis that eventually came to be, but I wasn't a part of it. The part I was part of was sort of before that. I was a true believer in this, and Comcast gave me an opportunity to try it out. It was just an experiment, but over the course of five years, we pretty much got rid of the old “throw it over the wall” way of doing application security to the shifting of ownership left onto the engineering team, which includes developers, includes DevOps people, includes site reliability engineers (SRE), includes platform engineering, etcetera.

While I was there, I had every tool, vendor under the sun. I can name some of the competitors of Contrast now: Checkmarx and Fortify and you name it. We probably had a little of it via acquisition or something at some point. And I had to sort of manage all that and I used measurement to try to figure out which teams had the most success in actually lowering our real risk. Not just checking boxes, but lowering our real risk, and Contrast was the best one. 

So when it was time to leave Comcast, I basically went out to the vendors and said, hey, any of you wanna hire me? I know your stuff. I have a name in the industry. And I got offers from a lot of them and I chose Contrast because I feel like it has the best opportunity to really do this shifting ownership left to the engineering team.

Alan:

Excellent. Alright. Let's fast forward to today, now. Ready? So shift left is a thing. I think most people out here have heard of it. Yeah. It's much more accepted. Software supply chain security is, you know, it kinda rolls off the tongue at a conference here at RSA.

But I learned this from you actually. Right? Instead of shifting left, do we have to shift everywhere? Right? Shift everywhere. Meaning that we can't take our eye off the ball of application security testing in production.

Larry:

We can't. We can't. It can't be at that expense that we're shifting left to developers and sort of the preproduction phase of it. We need to focus on the entire life cycle.

Alan:

I know it's kinda tied into where you're going. Yeah. Talk about it.

Larry:

Yeah. So I prefer shift smart rather than shift everywhere.

Alan:

Shift everywhere is sort of like, well, you know, if you do everything, we're not doing anything.

Larry:

Right? Exactly. You know, so I sort of like that better. But honestly, the thing that shifts left to me has never been the activities.

It's never been shifting left in this SDLC cycle. It's always been about shifting ownership away from specialists that are siloed to engineering teams that think holistically about delivering value to your customers. That's what DevOps is. 

So, you know, basically, you can't have DevOps without Agile, and you can't have shift left without DevOps. To me, there's this evolution of shifting ownership to the engineering team, including the DevOps people. And the DevOps people very much care about how the product performs in production. 

And let me give you a little example of go right to go left, which is kind of what you were hinting at, or the shift smart/shift everywhere kind of thing, and that is load and performance testing. 

You're old enough to probably remember when that was all done pre-prod. You had to stand up an environment that was similar to the current environment. You had to carve off a piece of your testing suite and run it multiple times: add load, just sort of see how the thing performs. But that was a simulated load. It wasn't a real load. It was a simulated environment.

It wasn't a real environment. And you still had problems when you got to production. Plus, it was very expensive to maintain and do all that work. It was awful.

And a lot of people opted out of doing it. The same thing that's happening now. They do AppSec really well. You have folks that have a lot they have to do. It's very expensive.

It's very difficult. And you have people that opt out of most of it or some of it, and they just do sort of the bare minimum that sort of checks a box and gets it done. Unfortunately, this is a theme in security. 

But with performance and load testing, an interesting thing happened. Along came technology and practices and mindset that sort of allowed you to actually shift load and performance testing from something you did pre-prod to something you actually can do safely in production. And the technology that was necessary were lightweight agents that didn't overwhelm the system and didn't consume too many resources. 

The practices include a lot of DevOps practices, but let's use canary deploys as the quintessential practice that sort of enables you to do that in production. It'll load and balance. So if you release just to one of your 100 servers and you see a load or a performance problem there, then you obviously can not distribute it to the other 99. That's the idea of that. And then the mindset of sort of being willing to sort of trust that this was an OK, safe thing to do, that had to come along, and it took a while for all of that to catch up.

Very few people do pre-prod load and performance testing. It's all done with canary. The people who were ahead of the curve and who had great load and performance testing pre-prod basically shut that down.  But you had people who were never willing to put the investment in, now doing that, now able to use these lightweight agents that do the testing.

Well, along comes Contrast. Contrast is a lightweight agent. We made it an order of magnitude less resource consumption since I've been there and then now we just released just a couple weeks ago a new version that's another order of magnitude lighter weight, and we can run that in production. 

And so now we can basically, you know, bypass a lot of the heavyweight things and our agent does not just do what a Static Application Security Testing tool would do, a SAST tool. It does also what an SCA [Software Composition Analysis] tool would do. It does also what a DAST[Dynamic Application Security Testing] tool would do, and it also does part of what a WAF [web application firewall] tool would do and part of what an observability platform would do. 

So the APM [application performance management] vendors are also getting into the security space. And we're coming at it from the security side and getting into the APM space, and they're coming at it from the APM space. And they don't even call it APM. It's the observability space for a lot of these guys. And look, make no mistake. Observability's security today. That's the focus of it. It always has been. They just never said it. But it's still security. Security was the engine, I think, that drove the APM train, if you will, in a lot of places.

You know, it would be because, what's the blast radius of an attack when you're doing incident response? If you don't have observability, you can't figure that out.

Alan:

Absolutely. And we've come a long way with it. Anyway, Larry, great stuff. People wanna get more information on Contrast Security? Where do we go?

Larry:

Yeah. Well, we were here at RSA. Please, you can connect with me directly on LinkedIn, my last name — Maccherone — is pretty unique. I'll be glad to set up a time to meet you for lunch or coffee or something like that. But we're at contrastsecurity.com as well.