Peiter Zatko's (Mudge) Cyber Independent Testing Lab methods.... just another flash in the pan?

Last week, The Intercept published an article (A Famed Hacker is Grading Thousands of Programs – and May Revolutionize Software in the Process) discussing a new method for testing and scoring the security of software. This new method – called the Cyber Independent Testing Lab – was developed by security expert Peiter Zatko, known more commonly by his hacker handle “Mudge.” According to the article, Mudge’s technique involves, in part, a static analysis of binary software files using algorithms to measure the security hygiene of code.

Point-of-View by Jeff Williams
Mudge's "cyber independent testing lab" methods.... just another flash in the pan?

I am of two minds about this effort.

On the one hand, this is great! I have been pushing for increased visibility into the security of applications forever. I made the OWASP mission “to make application security visible so people can make informed decisions about risk.” Note that this is not slavish addiction to security perfection, but an attempt to solve the fundamental information asymmetry between buyers and sellers in the software market. When the details are available to buyers then market forces can help us achieve Goldilocks Security (just right level of assurance). So I am 100% convinced that software security will be a problem until we create visibility. And CITL (assuming it works) could do this and help cure the “market for lemons” that we are in.

On the other hand, there are a number of practical problems. The first problem is that static analysis isn’t accurate enough to make good risk decisions. There are many false positives and (possibly worse) there are many false negatives. These could create either a chilling effect or false confidence, respectively – both are bad.

Even if you solve the accuracy problem, it won’t automatically work. Let’s say you started producing nutrition facts-style labels about products. Consumers probably won’t care very much. They have no way to sue for negligence, because software is licensed. And those EULAs have terms that limit liability for any software problems, including security. So you would need regulatory action, probably Congressional action, to create laws invalidating this type of contract clause. That ain’t gonna be easy because the software lobby has successfully fought that type of legislation for decades.

Nevertheless, if the program makes the labels public enough, they may eventually achieve the goal anyway. The idea is that even if buyers don’t care, the *sellers* may not want their products to have a terrible score on the label. I did a talk about this several years ago and I studied a broad range of labels.  It turns out that most labels, including the legendary nutrition facts label, mostly affect the sellers.  Because nobody knows what riboflavin is.

But all the claims about this program finally enabling software liability and cyberinsurance are just pipe dreams. There’s nothing really new here – Crispin Cowen started the Sardonix project in 2002 to do exactly this as an early crowdsourcing experiment. But it was quickly mothballed. Tavis Ormandy is doing this as part of Google’s Project Zero to great effect. But they can’t scale without compromising assessment quality 

The story of software security is about a broken market, very much like unsafe automobiles in the Ralph Nader era, and many people don’t see the complexity. This is about markets, economics, legal regimes, lobbying, culture and analysis technology. Maybe Mudge has lined up Congress, the software lobby, novel technology and consumers. But my gut tells me that this is just another flash in the pan. And I hate that.

 ~ Jeff Williams, CTO
Contrast Security Security