The GitHub Memcached DDoS: It shouldn't have happened

The record-breaking, 1.35 TB DDoS attack this past week against code repository GitHub, using Memcached servers—a few have sardonically labeled it Memcrashed—shouldn’t have happened.

Not in the sense that people shouldn’t do bad things to other people, like attack their websites, even though yes, of course they shouldn’t.

It shouldn’t have been possible

It shouldn’t have happened because it shouldn’t have been possible.

Supposedly everybody using Memcached servers knows they were never intended to be exposed to the public internet, since anyone can query them and they will politely respond, no authentication required.

Apparently not.

As Steven J. Vaughan-Nichols put it on ZDNet, “All too often, incompetent system administrators (sysadmins) have exposed memcached-enabled servers to the internet. Memcached was never, ever meant to be available over the public internet. It has no authentication, so it’s easy to abuse.”

Or as Sammy Migues, former principal scientist, with Synopsys Software Integrity Group, put it, “If you still think you must have your Memcached server exposed to the internet, you’re still wrong.”

And as Akamai, which provides a DDoS mitigation service called Prolexic, noted in a blog post this past week, when an attacker can spoof IP addresses of UDP traffic, “the protocol can be easily abused as a reflector when it is exposed to the Internet.”

By how much? “It is possible that an attacker could purposely place a 1MB value in the data store, and using a spoofed UDP packet request that single 1MB value hundreds of times per request,” Akamai said.

“This would result in a massive amplification factor where a 203 byte request results in 100MB response of reflected traffic, per request. It doesn’t take much imagination to see how this could be and is being abused.”

Significant attack surface

The current attack surface of vulnerable Memcached servers is significant.

Akamai reported 50,000 servers, but ZDNet reported finding 91,000 with a quick Shodan search.

But if there is any good news about the attack—and there is—it is that the damage to GitHub was minimal, and preventing further attacks like Memcrashed shouldn’t be too difficult or expensive.

Indeed, within 10 minutes of the start of the attack, GitHub called for help from Prolexic, which intervened, routed all traffic to its datacenters, and then detected and blocked malicious packets. Eight minutes later, the assault dropped off.

Compare that to the then-record Mirai botnet-enabled DDoS assault on New Hampshire–based internet backbone provider Dyn in October 2016, which took down major players like Twitter, GitHub, PlayStation, Netflix, Reddit, and a host of other sites for much of that day.

Of course, that kind of protection doesn’t come cheap. While nobody is saying specifically what it costs, it has to be significant. Recall what was then a record-breaking 620 Gbps DDoS attack against security blogger Brian Krebs in September 2016. Akamai was providing DDoS protection for Krebs pro bono at the time but said after the attack, which was double the volume of anything it had seen before, it could no longer afford to do so.

Krebs was understanding. “I can’t really fault Akamai for their decision,” he said at the time. “I likely cost them a ton of money today.”

No botnet required

Memcached DDoS attacks are a different animal, of course—no botnet required.

As described by Practical Technology, Memcached is an “open-source, high-performance, distributed, object-caching system. It’s commonly used as an in-memory key-value store for small chunks of arbitrary data.”

While it’s best known for being used by social network platforms, it’s attractive to many others as well because it speeds things up—it sends any program’s first attempt to retrieve data to its cache before querying the database.

All caches are filled with recently retrieved server data, so when a user asks for data, Memcached provides it from fast memory, rather than the slower hard-drive-based database.

Hence its popularity.

But it also doesn’t question requests—hence its vulnerability as well.

Easy to exploit, easy to fix

But while the current situation is easy for attackers to exploit, it is also relatively easy to fix. “It falls squarely into the category of ‘Doctor, it hurts when I do this,’” Migues said.

And he provided the simple steps necessary to do so:

• Make sure your Memcached systems are not exposed to the internet.
• In every perimeter firewall you have, immediately block all access from the internet to UDP port 11211 (or wherever, if you mistakenly thought you were being clever by running your server on a different port).
• Disable UDP on all Memcached servers.

More generally, “ISPs need to block spoofed packets from exiting their networks, and protocol developers need to better understand velocity checking and amplification attacks,” Migues said.