The Guerrilla Guide to Buying an Application Security Tool

If you're going to buy an application security tool, don't get distracted by hype.
Purchasing an application security tool is an important decision and it's easy to lose focus on exactly what you want out of the tool. We believe the goal should be to gain real assurance across your entire application portfolio.That means verifying that:

1. All the right security controls are present.
2. Those security controls are correct.
3. Those security controls have been used properly in all the right places.

When you're considering different application security tools, try to stay focused on how well they will help you achieve that goal.

There are a number of different approaches to application security, e.g. static, dynamic, interactive, & manual. But how should you choose?  In this article, we'll explore the key questions you should ask your vendor. Questions like:

• Does It Find the Vulnerabilities You Care About?  Seems obvious, and everyone claims to find the OWASP Top Ten. But the reality is that tool vendors often claim coverage even if they only find one tiny part of an OWASP Top Ten category. Consider all the scanning tools that claim to find weak encryption - are they really being honest about what they can find? Do you care about authentication and access control?  If so, you better find out whether your tools can help with these.

• Does It Cover All the Code In Your Application? Again, this seems obvious, but traditional tools just don't do a very good job at this. Static analysis (SAST) tools only look at your custom code and ignore all your libraries and frameworks. They also only explore your code from entry points that they understand. Not only does this leave significant parts of the codebase unassessed, it also creates many "lost sources" and "lost sinks" that lead to false positives. Dynamic analysis (DAST) tools have even worse coverage, typically exercising only a small fraction of an application. This is due to the difficulty of crawling complex applications with accounts that can invoke all the features of an application.

• Is It Easy To Setup and Use? Caveat Emptor, Latin for "Let the Buyer Beware", is often the mantra for SAST and DAST tools, especially because traditional static and dynamic tools are expert-only tools. You have to be an application security expert to install, configure, run, and use them - and especially to triage the results.  If your plan is to have developers run the tools themselves, you may want to test whether they are capable. It shouldn't be difficult to pick a real application and see if a developer can handle the tool and get good results.

• Is It Accurate? Accuracy is all about mistakes and misses. Mistakenly identifying a vulnerability, also known as a false positive, costs time. On average it takes an expert 10 minutes to diagnose a false positive static analysis result. At that rate, it takes a week to clear 240 false positives without making anything any more secure. Missed vulnerabilities are insidiously dangerous because they represent exposures you don't know are there. You may even have a false sense of security from a tool that misses almost everything.

• Does It Make Custom Rules Easy?  This isn't one of those times when not customizing is okay. You will absolutely miss a large percentage of the most critical vulnerabilities if you don't customize to your enterprise. Period. The reason is that every organization has their own patterns for authentication, authorization, validation, and encryption. These are the four horsemen of the security apocalypse. And you can stop them. But you have to customize to do it.

• Does It Provide Results in Real Time? Modern software development moves fast. Developers need security feedback immediately or the opportunity for learning is lost. Agile and DevOps projects can't wait for a periodic scan to get scheduled and analyzed. Developers should get real time security alerts on the code they are working on. If your tool requires an expert to schedule, configure, run, or triage, it will surely become a bottleneck that drives development teams crazy.

• Can It Scale to Your Entire Portfolio?  For this one, try a little math. Multiply the number of applications in your portfolio by the amount of time it will take to install, onboard, run, and triage for an average application. Then consider the number of people you have to do this work. In most organizations, anything but a continuous, automatic solution won't work. Look for the enterprise analytics that will allow you to make strategic decisions about application security, instead of the one-app-at-a-time patching model.

• What Is the Pricing Model?  Consider what is easiest for you to manage: Do you want a per-seat model where you have to figure out how many developers, architects, security, and compliance people need access? Do you want to calculate the size of your apps and pay by the line of code? Or would you prefer to pay one price per application, with unlimited seats, lines of code, and instances of the application?

When it comes right down to it, you've got options when you are shopping around for application security tools. You can go with big companies who acquired 2002-era SAST and DAST technology through buyouts of smaller firms (e.g. HP Fortify or IBM AppScan) or you can go with niche companies who know what they're doing and get the job done right (e.g. Contrast Security). Yes, as you can probably tell, we've taken sides.

So there you have it - the guerrilla guide to buying an application security tool. Have additions? Questions? Comments? Leave them for us in the comment section below. Thanks for reading.