The cathedral and the bazaar of software security vulnerabilities

In 1997, Eric Raymond published a now-famous essay entitled “The Cathedral and the Bazaar,” which was exploring two different schools of thought related to software development. Nearly 20 years later, this same tension between hidden and open development continues to play out in application security.

The Wikipedia article about the essay summarizes the fundamentals. “Cathedral” style software is centralized, somewhat hidden from view and released when blessed from on high by authorities. “Bazaar” style software is developed in the open, rapidly trying and rejecting approaches, and settling on solutions over time through consensus.¹

The advantages of bazaars in security

In his essay, Raymond introduced what he called “Linus’ Law”—that “Given enough eyeballs, all bugs are shallow.” The premise is that the more users software is exposed to, the more readily the bugs are found. Diverse users will stress systems in diverse ways, and that stress reveals bugs that no small set of users—even the developers themselves—would find as easily or as quickly. One can see HackerOne, Bugcrowd and similar offerings as almost literal bazaars, allowing all comers to look for security bugs under well-defined and regulated conditions.

Bazaars have some clear advantages. Outsiders test software fresh, with fewer preconceptions and assumptions than the vendor themselves. They may also have experiences and tools that a software vendor’s team lack. Economics seems to favour this model also, because the increase in supply of testers and test effort should drive the price of test effort down. These arenas also reduce some tedious bureaucracy that comes with security testing, like signing up to test different environments and knowing where to report findings.

The advantages of cathedrals in security

In this mindset, Mary Ann Davidson’s deleted blog post is very clearly written from the highest room in the tallest tower in the cathedral. Her view, which is shared by many in similar positions, is that vendors know their systems best. The arrogance of the article borders on hostility, assuming that no one else could possibly find security bugs valuable enough to merit Oracle dealing with an outsider (or—gasp!—giving credit to them). There are, however, true advantages to the cathedral approach that might be overlooked in the almost cartoonish zeal of Davidson’s post.

It’s strictly true that a vendor like Oracle, full of smart people and decades of history, will know its customers and its products better than anyone. They’ll be able to understand the ramifications of a security patch far better than any outsider. And it’s probably true that they have the best diagnostic tools for instrumenting systems, executing test cases and analyzing results. Any fix to a product as sophisticated as Oracle will undergo rigorous testing beyond what any outsider can possibly simulate. Oracle engineers are probably jaded from absurdly simplistic reports from well-intentioned individuals that boil down to, “The system doesn’t validate input. It should validate input.” This is vapid and useless advice to a vendor. I think outsiders bring much less to the table with respect to fixing bugs than the engineers at a vendor.

Why the world needs cathedrals and bazaars

Raymond quotes Torvalds as saying, “Somebody finds the problem and somebody else understands it. And I’ll go on record as saying that finding it is the bigger challenge.” In security in 2015, with the advent of the bazaar of crowd-sourced security research and many bug finding tools, I believe finding bugs is now the easy part by a large order of magnitude. Understanding the problem fully and knowing the right fix is the rare commodity now. Rarer still (and perhaps the subject of a future blog post) is the know-how to prevent them in the first place.

At Black Duck, we believe that application security is fundamentally about “identify, remediate and prevent.” The bazaar is an excellent way to find bugs and the cathedral is the right way to get the fixed software into the hands of users. We have observed in the BSIMM a few large and sophisticated software security initiatives that incorporate bug-bounty programs. We don’t say whether they should or shouldn’t. As the BSIMM, we merely observe and report the fact that they do.