Interview: Jonathan Chow & Neeta Maniar of Live Nation Entertainment

In this interview, Jeff Williams interviews Jonathan Chow and Neeta Maniar of Live Nation Entertainment: The world's leading producer and promoter of live entertainment, and the parent company of Ticket Master and The House of Blues. They discuss issues pertaining to SDLC, security organization and management, and how to ultimately be more secure.

You can LISTEN TO IT on SoundCloud

Jeff Williams: Hi Everyone. I'm Jeff Williams of Contrast Security and I want to thank you for joining us on our episode of The Security Influencers Channel. Our goal is to provide a series of brief and hopefully highly informative interviews with and for security professionals.

Our theme for 2014 is discussions on the implications of continuous. Today we're talking with Jonathan Chow and Neeta Maniar from Live Nation Entertainment: The world's leading producer and promoter of live entertainment, and the parent company of Ticket Master and The House of Blues.

Jonathan is the Senior Vice President and chief security officer for Live Nation and Neeta is the Senior Director of Application Supplier Assurance at Live Nation and they're both charged with running big application security program.

The first question I have for you is: what's the one thing that deeply bothers you about the way that people practice application security today?

Neeta: I'll take that one.  One of the things that really bothers me is that we're so focused on finding vulnerabilities versus it's more of a reactive approach than proactive and where I want to be spending most of my time is in educating our developers and getting them to build code right the first time around, and yet I spend most of my time just trying to justify why we need to fix the flaws that we found. I think for me-and then also the fact we're finding vulnerabilities that existed 10 years ago, like sequel injection, cross-site scripting these are all the things that hackers are using these days and we're still not getting good at fixing.

Jeff: So what do you do to justify the program?

Neeta: Well I try too-we try to use industry examples, we try to say "look this is what happened to Target, this is what happened with various other big companies that have experienced data loss or they've had a major applications security breach." And say "Look this could happen to us as well." Right? And basically security is and should be part of the functioning operation of the application, it shouldn't be a nice to have anymore not in this day in age for sure.

Jeff: Yes. Right. Jonathan?

Jonathan: What Neeta said is absolutely true. I've been involved in part of an applications program here for 12 years now, and we're still having developers here are creating the same flaws that my previous job they're creating the same flaws across the industry and so the education piece I think is what's missing to stop. We're not discovering anything new. There's nothing new that our testers are finding. It's the same thing. It's not getting better and that's the part that really bothers me more than anything else.

Jeff: I couldn't agree more actually. I wrote the first version of the OWASP top ten in 2002, and it's essentially the same stuff in there still after you know 12 years. It's really not changing, so that's a bit of a failure for the security industry, really, I think.

So next question. How do you guys stay on top of your portfolio of applications and developers writing new code and new vulnerabilities coming out and new vulnerable libraries coming out, how do you stay on top of all of that from your perspective?

Jonathan: It's almost a job all unto itself. Especially a company, any big company there's not just what you know about, but there's all the rogue activity that happens. The lunch time deals, oh create a website or create a function and all of the sudden it's up and running at Go Daddy by afternoon. So it's really tough but I think maintaining good relationships with your business partners is one way to try and combat that.

Jeff: Interesting. Why with your business partners? Do you mean with you developers? Or with external entities?

Jonathan: Actually external entities. I think that information security programs are traditionally are really focused on the relationships they have with the tech teams, so whether it's a development shop or an IT shop, and I think that's part of the failure, right, because I think that the business in some cases with, see I can do this cheaper, faster, better, than if I go to the approved IT folks. They'll go outside and I think that's the primary driver for rogue work happening, and if you have a good relationship with the business and not just the IT folks and then hopefully they'll come tell you about that whether it's approved or not.

Jeff: Right. So Neeta you're in charge with the applications themselves, awfully fast rate of change going on there. How do you stay on top of it?

Neeta: Well I try to stay less application centric and more team centric. Right? So…Like Jonathan was saying you have to build good relationships and kind of trust with the different IT teams and the product teams and it's no longer just about the developer, it's about the process and getting in touch with the product owners. You know the key way engineers and architects finding out when you can insert yourself to get the right level of visibility when there's a new change.

We've actually just hired what we call business security leaders, so they're our liaison between what's happening in the businesses and the change their doing and the services that we provide. So we're just trying to make ourselves more visible in those areas. We' definitely don't have the bandwidth to be sitting in on every single meeting and making comments about every change, but it's part of that education thing where we're trying to empower the teams to do that better themselves.

Jeff: Interesting. I like that. You know I've been studying the ways that industrial factories monitor their complex systems. I've been learning about all the instrumentation that they use and how they bring all that data together and monitor it in real time. I just read a book called The Phoenix Project by Gene Kim who's one of the leaders in the DevOps movement and it's fascinating to me the approach of sort of comparing development shops to real factories, real industrial plants and stuff. I'm trying to learn from their experience. What I'm wondering-it sounds like what you're doing is actually, is almost like a human instrumentation where you're gathering data through relationships with all these various teams. What do you see happening in terms of automated instrumentation of the kinds of tools you were talking about earlier? Things like Jenkins and Eclipse and SPN and so on?

Neeta: I think it's really important for whatever, in the world of App security there's scanning technology, right? And it's important for any of that to be well integrated into the tools we already use. Any SDLC process whether you're doing QA or Builds and trying to inject security into those particular tools is going to be important for any instrumentation.

Jonathan: I think the analogy to kind of a manufacturing line is really interesting. You think about the process of car too. The process it takes to build a car for example. There's doors and windows and nuts and the engine and all sorts of electronics and you would hope that the quality is built into every step of the process and not just when it's all put together the car rolls out and somebody looks over and says "is everything working and did we put it all together right?" if they find a flaw at that point they have to take it all apart and I think that the approach to security, the movement towards, that we've been trying to push is the same thing, where you're checking at every step, you're embedding at every step because once it's done it's too late to take it apart again.

Jeff: You guy's remember that commercial where the car rolls off the line, I can't remember if it was a BMW or Mercedes but the they got a marble that rolls down to prove that the tolerance is just perfect? I'd love to be able to do more of that with our software. So let me ask you Jonathan, how do you feel about your visibility into the apps and other systems that you guys run? I see a lot of organizations who are doing an annual pentest they're doing a scan on some periodic interval but I feel like when you look at the portfolio level the visibility is really spotty. What do you do the fill in the gaps and make it look up-to-date?

Jonathan: A comment Neeta said earlier, was not enough Bandwidth. It's true for every IT security shop that I've ever talked to or been a part of. There's more of them than there are of us so it's impossible for us to get around to every application and when you focus on the application, you're always going to be overwhelmed. You're always going to be outnumbered. So when it comes to-something that we were talking about before about the education and making them want it and then providing them with the right tools, so they're not dependent on us, they're not waiting on the security team to do something for them. They can try to make it a self-service as possible so our approach is to generate the demand on their side to say "hey the security is something good and you want it." And then give them the tools, so they can do it without us getting in the way.

Jeff: That strikes me as exactly what needs to happen is the security experts really need to get out of the way and enable the development teams to do these things for themselves with automation and guidance and training and so on. Too long the security expert has been the bottle neck, and that's hard for me to say because that’s where I put myself.  But it's time to get out because it doesn't scale. It doesn't scale.

Jonathan: We still have expertise to offer in terms of providing the guidance and showing them at a macro level what to do, but you really do become the bottle neck when it comes to individual test. "I'm waiting for you on a test." That's-it's-at one point 10 years ago that was important because we were the only ones who could run the tool or interpret the tool but I think that the industry has to evolve to empower the people, who have accepted the fact it's important to them, that's fine. Make it a part of their jobs.

Jeff: I call that old approach, “Security that pisses everybody off!”, because nobody's happy right?

Neeta: That's an interesting point because I remember working at GE and having that-you'd have such a long time between when an application requirement came out and when it was released. I mean it could be a year or two years before an application came out and you had all this time to do the review and talk through remediation with the teams, and here at an agile environment, if you're not there then you miss it and it's kind of harder now to have that position.

Jonathan: It's actually the worst of all worlds, if you miss it because then they want something from you, you either slow them down and then they won't come back or you interrupt their process and they see you as incompetent. The development world has evolved to a point where the new methodology they are using is very, very fast, and the securities industry hasn't kept up. So we risk becoming that proverbial dinosaur where we don't have a place in the new world.

Jeff: I think that is a huge danger and I do see security groups getting cut out of the loop in a number of instances. Do you feel that that's the only pressure on security groups? The move to Agile and then even DevOp's kind of organizations, is that the major thing that changed? Or are there other things that are changing, or changing the way that people do security or information security?

Neeta: I think there's also a positive change. I think that application security is a pretty hot topic now, more than it was years ago, it's more visible. We joke that we use security breaches that happen as our leverage to try and convince teams to do more--and we have more fuel for our story, and to say, "Look! This is why you need to do things." I think that's more of a positive change. As far as the challenge for us it's really just having the bandwidth to keep track of everything that's going on and prioritize what-choose your battle type of thing.

Jeff: I know we've broken out of the echo chamber, when my mom calls and says "What's going on with this HeartBleed thing?" So let me finish up with one last question. I want to know what the metrics are, the key metrics that you want to know, or write so you can sleep at night. So Jonathan let's start with you.

Jonathan: So that I can sleep at night. Obviously a raw number of flaws in applications is a key metric for me. For me I think a really telling metric would be by team or in situation of some companies where they outsource to other companies and being able to measure the quality of what you're getting back. Whether its flaws for lines of codes delivered. I would love to get down to the point where I can go to a specific developer and say "You know you've been making cross-site scripting errors since 2006, you've made it here and you've made it in January here, you made it in March here, you made it in October here, I need to teach you something. We're not at that level, but that's trying to make us better.

It's not just pointing out issues. I think that if we can get to that point where the developers and development teams and outsource development shops can accept the fact that securities teams are here to make them better at their jobs and not just to point out what's wrong, then I think that it will gain more momentum.

Jeff: That's great. That's perfect. Neeta, do you have some metrics that you really like?

Neeta: Oh so many. I think that any metrics that help us understand the progress, like trending metrics, from point A to point B, if you run X number of scans have the vulnerabilities gone down over time or if you have a score, I think that's been really helpful for us to say to a team you know "Congratulations! Your score improved." And to be able to give that positive reinforcement, and then on the educational side, vulnerabilities by technology just so that we can figure out, what should we be training our teams on?

Jeff: How often do you scan applications? Like for a typical supper project. Like how often are they getting their code scanned?

Neeta: Well I have one team that's running a scan every two weeks or so, like it's basically depending on how often the application is changing so they pace their scanning with the rate of change in the application and that's for automated scanning but manual assessments, we barely, we maybe do one a year. There's definitely a need for both, but the automated scanning gets run more frequently and then the manual scanning, as soon as we get our hands on the application in a stage environment.

Jeff: Thank you so much for sharing a little bit about how you guys do security and applications security.