Cloudbleed, like Heartbleed, may affect millions

A researcher from Google disclosed on Thursday that private messages, API keys, and other sensitive data were being leaked by a major content delivery network to random requesters, a leakage that could affect up to 5.5 million websites.

Like Heartbleed, which was co-discovered by the Black Duck team in Oulu, Finland, and Google in April 2014, a new vulnerability dubbed “Cloudbleed” was discovered through routine fuzz testing. The researcher, Tavis Ormandy, the bad boy of vulnerability research at Google Project Zero, said that on February 17, 2017, said he noticed the leakage of private session keys and other sensitive information across various websites in his results and quickly isolated the problem to those sites hosted by Cloudflare, a content delivery network (CDN) and web security provider.

“I encountered some data that didn’t match what I had been expecting,” Ormandy wrote. “It’s not unusual to find garbage, corrupt data, mislabeled data or just crazy non-conforming data…but the format of the data this time was confusing enough that I spent some time trying to debug what had gone wrong, wondering if it was a bug in my code. In fact, the data was bizarre enough that some colleagues around the Project Zero office even got intrigued.”

Following Project Zero’s seven-day policy for actively exploited attacks, Ormandy made public his findings on Thursday.

The problem

“It looked like that if an html page hosted behind Cloudflare had a specific combination of unbalanced tags,” Ormandy wrote, “the proxy would intersperse pages of uninitialized memory into the output (kinda like Heartbleed, but Cloudflare specific and worse for reasons I’ll explain later).”

Like Heartbleed, which is a vulnerability in the implementation of the heartbeat feature in OpenSSL and not a flaw in SSL itself, Cloudbleed is limited only to customers. The company boasts, however, optimization for over 5.5 Million websites due in part to a free program that allows almost anyone to sign up and use. And like Heartbleed which existed in the wild for nearly two years, Cloudbleed may have leaked sensitive data from websites using its services between September 2016 and February 2017 onto third party search engine and data caching sites worldwide.

Mitigation of Cloudbleed

Cloudflare, to its credit, gave priority to this bug. Working around the clock from two command centers, one in San Francisco and one in London, the company ascertained the problem and proposed a workaround within 3-4 days.

As noted in their company blog, Cloudflare traced the leakage to a move away from their Ragel-based parser and started using a new parser, named cf-html. In doing so, “our edge servers were running past the end of a buffer and returning memory that contained private information such as HTTP cookies, authentication tokens, HTTP POST bodies, and other sensitive data.” As a workaround, the company disabled three Cloudflare features–email obfuscation, server-side excludes, and automatic HTTPS rewrites–which contained the bad cf-html implementation.

The company adds, “For the avoidance of doubt: the bug is not in Ragel itself. It is in Cloudflare’s use of Ragel. This is our bug and not the fault of Ragel.”

Danger remains

The danger here is in the amount of cleanup necessary. There is data on search engines and data caching servers worldwide. It is a race against the clock to purge this confidential data.

Cloudflare says the greatest period of impact was from February 13 to 18. At that time around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulted in memory leakage. To put that in perspective, the company says that’s about 0.00003% of all requests during that period.

Consumers may not necessarily use Cloudflare yet the sites they use may. Here’s a list of sites potentially affected by the leakage. It is important to remember that the sites listed do not necessarily mean they are vulnerable to Cloudbleed, only that they use the service.

What to expect from Cloudbleed

Researchers on Friday note that mobile apps may also be affected. NowSecure has identified nearly 200 iPhone apps that use Cloudflare. Meanwhile, 1Password, which uses Cloudflare, said it is not affected. Expect more of this in the coming days as individual companies begin to identify whether their products and services are affected.

And consumers should expect this reporting on Cloudbleed to continue for months. Heartbleed is nearly three years old, yet last April researcher Billy Rios found that out of the original 600,000 IP addresses original infected, some 200,000 remained vulnerable 48 months later.