Attackers have seized on a relatively new method for executing distributed denial-of-service (DDoS) attacks of unprecedented disruptive power, using it to launch record-breaking DDoS assaults over the past week. Now evidence suggests this novel attack method is fueling digital shakedowns in which victims are asked to pay a ransom to call off crippling cyberattacks.
On March 1, DDoS mitigation firm Akamai revealed that one of its clients was hit with a DDoS attack that clocked in at 1.3 Tbps, which would make it the largest publicly recorded DDoS attack ever.
The type of DDoS method used in this record-breaking attack abuses a legitimate and relatively common service called “memcached” (pronounced “mem-cash-dee”) to massively amp up the power of their DDoS attacks.
Installed by default on many Linux operating system versions, memcached is designed to cache data and ease the strain on heavier data stores, like disk or databases. It is typically found in cloud server environments and it is meant to be used on systems that are not directly exposed to the Internet.
Memcached communicates using the User Datagram Protocol or UDP, which allows communications without any authentication — pretty much anyone or anything can talk to it and request data from it.
Because memcached doesn’t support authentication, an attacker can “spoof” or fake the Internet address of the machine making that request so that the memcached servers responding to the request all respond to the spoofed address — the intended target of the DDoS attack.
Worse yet, memcached has a unique ability to take a small amount of attack traffic and amplify it into a much bigger threat. Most popular DDoS tactics that abuse UDP connections can amplify the attack traffic 10 or 20 times — allowing, for example a 1 mb file request to generate a response that includes between 10mb and 20mb of traffic.
But with memcached, an attacker can force the response to be thousands of times the size of the request. All of the responses get sent to the target specified in the spoofed request, and it requires only a small number of open memcached servers to create huge attacks using very few resources.
Akamai believes there are currently more than 50,000 known memcached systems exposed to the Internet that can be leveraged at a moment’s notice to aid in massive DDoS attacks.
Both Akamai and Qrator — a Russian DDoS mitigation company — published blog posts on Feb. 28 warning of the increased threat from memcached attacks.
“This attack was the largest attack seen to date by Akamai, more than twice the size of the September, 2016 attacks that announced the Mirai botnet and possibly the largest DDoS attack publicly disclosed,” Akamai said [link added]. “Because of memcached reflection capabilities, it is highly likely that this record attack will not be the biggest for long.”
According to Qrator, this specific possibility of enabling high-value DDoS attacks was disclosed in 2017 by a Chinese group of researchers from the cybersecurity 0Kee Team. The larger concept was first introduced in a 2014 Black Hat U.S. security conference talk titled “Memcached injections.”
DDOS VIA RANSOM DEMAND
On Thursday, KrebsOnSecurity heard from several experts from Cybereason, a Boston-based security company that’s been closely tracking these memcached attacks. Cybereason said its analysis reveals the attackers are embedding a short ransom note and payment address into the junk traffic they’re sending to memcached services.
Cybereason said it has seen memcached attack payloads that consist of little more than a simple ransom note requesting payment of 50 XMR (Monero virtual currency) to be sent to a specific Monero account. In these attacks, Cybereason found, the payment request gets repeated until the file reaches approximately one megabyte in size.
The ransom demand (50 Monero) found in the memcached attacks by Cybereason on Thursday.
Memcached can accept files and host files in temporary memory for download by others. So the attackers will place the 1 mb file full of ransom requests onto a server with memcached, and request that file thousands of times — all the while telling the service that the replies should all go to the same Internet address — the address of the attack’s target.
“The payload is the ransom demand itself, over and over again for about a megabyte of data,” said Matt Ploessel, principal security intelligence researcher at Cybereason. “We then request the memcached ransom payload over and over, and from multiple memcached servers to produce an extremely high volume DDoS with a simple script and any normal home office Internet connection. We’re observing people putting up those ransom payloads and DDoSsing people with them.”
Because it only takes a handful of memcached servers to launch a large DDoS, security researchers working to lessen these DDoS attacks have been focusing their efforts on getting Internet service providers (ISPs) and Web hosting providers to block traffic destined for the UDP port used by memcached (port 11211).
Ofer Gayer, senior product manager at security firm Imperva, said many hosting providers have decided to filter port 11211 traffic to help blunt these memcached attacks.
“The big packets here are very easy to mitigate because this is junk traffic and anything coming from that port (11211) can be easily mitigated,” Gayer said.
Several different organizations are mapping the geographical distribution of memcached servers that can be abused in these attacks. Here’s the world at-a-glance, from our friends at Shadowserver.org:
The geographic distribution of memcached servers exposed to the Internet. Image: Shadowserver.org
Here are the Top 20 networks that are hosting the most number of publicly accessible memcached servers at this moment, according to data collected by Cybereason:
The global ISPs with the most number of publicly available memcached servers.
DDoS monitoring site ddosmon.net publishes a live, running list of the latest targets getting pelted with traffic in these memcached attacks.
What do the stats at ddosmon.net tell us? According to netlab@360, memcached attacks were not super popular as an attack method until very recently.
“But things have greatly changed since February 24th, 2018,” netlab wrote in a Mar. 1 blog post, noting that in just a few days memcached-based DDoS went from less than 50 events per day, up to 300-400 per day. “Today’s number has already reached 1484, with an hour to go.”
Hopefully, the global ISP and hosting community can come together to block these memcached DDoS attacks. I am encouraged by what I have heard and seen so far, and hope that can continue in earnest before these attacks start becoming more widespread and destructive.
Here’s the Cybereason video from which that image above with the XMR ransom demand was taken:
its terabits not giga … you are off by factor of 1000
That’s what I read earlier, also…
Are these scammers asking for payment via check? If so, I would like to know if there is a location in Oregon where scam victims are asked to send money to…
Read the body of the text.
“I always do that. I always mess up some mundane detail.”-Office Space (1999)
Interesting that DHS US-CERT just emailed subscribers February 27 with an article about this:
https://www.us-cert.gov/ncas/alerts/TA14-017A
Interestingly, one of the hosting providers listed above is one that I use, which has now blocked me out of SSH and the administrative section of one of my websites after reporting that my website exceeded it’s database quota. Which doesn’t make sense, as it prevents me from fixing the problem. The date of the blockout and alleged database problem: February 26, 2018…
One of Akamai’s clients? Let’s just say it was GitHub: https://githubengineering.com/ddos-incident-report/
(Hey Wlad.)
Why aren’t the offending memcached servers just kicked the heck off the net in the meantime while we figure out how to reduce their exposure? Is there no way to intercept memcached requests and cull off the ones with large multiplication values? Do we even “need” memcached as a critical function of any major service, or can they be accomplished through other DB functions (albeit slower..) ?
There are a bunch of factors involved:
1. a given computer assigned to a given IP address is paid for by someone under a certain contract with a Hosting provider.
— If the contract doesn’t have a provision to disconnect a computer/IP for excessive traffic (it probably bills for traffic), then it’s hard to disconnect it, since that’s a breach of contract.
2. a given Hosting provider doesn’t have sufficient monitoring / reachable people, then it’s hard to even get to this.
3. a Hosting Provider has a contract for transmitting data to an ISP — If the contract doesn’t have a provision to disconnect a computer/IP for excessive traffic (it probably bills for traffic), then it’s hard to disconnect it, since that’s a breach of contract.
4. the ISP has contracts with Peering providers — If the contract doesn’t have a provision to disconnect a computer/IP for excessive traffic (it probably bills for traffic), then it’s hard to disconnect it, since that’s a breach of contract.
5. Generally each layer past the hosting provider doesn’t want to do much packet inspection, because that would slow down traffic…
6. The entire system is built on trust.
It’s problematic for anyone along the chain to block traffic.
Interesting to me is that the list of top 20 ASNs contain many that we have banned from sending traffic to our client’s sites:
OVH
DIGITAL OCEAN
AWS
LINODE
ENZU
CHOOPA
EGI HOSTING
NOBIS
QUADRANET
We may be banning some of the others but I don’t recognize the names.
These networks earned their place on our banned list by being confirmed sources of email and contact form spam as well as PPC click fraud and other abuse. Most of you could probably ban or block all them from port 80 and 11211 without any ill effects to your operations unless you need international traffic.
Anyone that would like a copy of our htaccess block list can have one. It’s taken us several years to compile by hand, but if it will help others I’m glad to share it.
Maybe put it on non-public github or equivalent?
Umm, how? You forgot to provide an address.
Just hover your mouse over his name and read off the address from the status line. A lot of posters to the BK comments section give a website address.
Just returned to read the comments. Sorry, I thought security experts would notice one of my domains under my name and know how to contact me that way. 🙂 I also thought my email would be linked but I see that it’s not. Sorry. You can contact me via nielsentech@gmail.com. I already have two requests for the htaccess Deny List.
But please note: This list is hand-compiled and may have errors that could block something you don’t want to block. Also, the list is designed to block most NON-USA traffic and traffic from hosting company IP ranges and data centers that do NOT provide residential service. Most ranges have shown to be sources of spam, bot traffic, and PPC click fraud. Use at your own risk and if you don’t understand some of this, get some help (but not from me. Questions ok, user support not ok.).
Bill, you might want to fix the “…is a douchebag…” comment one notices when clicking your name and going to your website.
OK, I don’t get all of it. But IF this is an easily-detected and blocked DDoS, isn’t the headline a bit overdone? Looks like something from KnowBe4.
Is ddosmon.net reputable? I hate to stereotype but the .cn domain is concerning (yeah, I know they could have chosen an address with .com and I might not notice it).
Can anybody recommend another service to alert when a specific IP gets attacked?
As someone mention above: “the entire system is built on trust.” That’s the real problem. It’s led to is the ability to spoof IP addresses. In this case, it’s the requester’s address that is spoofed. Some other DDos methods rely on spoofing to hide the source of the attack.
It’s going to take new IP traffic protocols, which are not built on trust, to solve the spoofing problem. I have no faith that egress filtering or transit filtering by ISPs and network operators will ever happen voluntarily.
Ya – back in the day there were only the good guys (us) and the bad guys (the Soviets). World is very different now. Still, check VOX-POL.
I’ve never really understood what the ultimate issue is with last mile ISPs putting rules in place stating if the source address on an outgoing packet isn’t from one of their subnets to route that packet to /dev/null. Hosting providers could do the same at the edge routers of their datacenters. They don’t even need to be super specific about which IP ranges are actually in that particular center, any IP range used by any center can go outbound with a source address from one of their IPs. Same with the ISPs. But but but VPN providers, well guess what, they could do it too. The closer to the customer they go, the less horsepower required because the fewer packets that will be /dev/nulled.
I’ve done it at numerous employers at my edge routers with zero consequence on traffic that legitimately should be passed. Hell, canning email where the from: and to: are the same has far more of an impact, since some rather large companies craft their bulk messages to look that way for reasons that make sense only to them.
I get it, they don’t want to do it because someone in marketing has them implementing features they don’t need and beancounters in accounting have cut their hardware budget so deeply they can’t realistically implement marketing’s features much less anything else.
You really don’t understand the interwebz, do you?
There is an ingress filter RFC (2827) that has been around since 2000, but the problem is, almost no service providers implement because they are worried about the impact to their network.
It is a circular argument. “I’m not going to implement ingress filtering because if I do and no one else does, it will bring my equipment to its knees.” Once the DoS, DDos, DrDos traffic makes it past the first hop from the source, the amount of traffic goes up quickly.
Because no one is implementing this RFC, datacenter owners now have to pay providers with the infrastructure to filter this kind of traffic at the destination end in order to meet their BC requirements. They basically have to pay for a failsafe that should be configured into the internet.
It’s long overdue that ISPs should CEASE PEERING with any other ISP that fails to implement BCP38/RFC2827 (published almost 18 years ago!) Adhering to RFC 2827 (https://tools.ietf.org/html/rfc2827), section 3, would probably come close to eliminating this type of UDP amplification attack.
Right – but ISPs (like most folks/companies) are in there for the money nowadays. Nobody cares about these kinds of attacks, just like nobody (really) cares about climate change…
They can’t see that if they’re unable to peer they’d start losing money? ISP’s are still dependent on others to make their business go.
That might be true. How about a more “enforcement” based approach? What if the IANA, or ARIN (et al), were to augment their review process on address-block allocations, requiring block holders to show that they’re adhering to RFC 2787, and impose sanctions if they’re not? Say, you get X-time-period to clean up your act, or we claw back your IP blocks? Hard to make money as an edge-provider ISP when you have no IP addresses to work with.
This may be a naive question, but if the attack vector is that you can make a GET request to a memcache server and let it send a large amount of response data to a spoofed IP, then how is that different than making a HTTP request for a large file (an ISO for example), and letting the web server send that response to a spoofed address? Thus making web servers that serve big files just as dangerous as memcache servers, in terms of DDOS? I’m sure there’s some difference I’m not aware of, I just don’t know enough to know what it is… thx.
This type of attack is an “amplification” attack. You, the attacker, send a small request to an unwitting server that has a big pipe, and fool it into sending its large response to the intended victim by stuffing the victim’s IP address into “source IP” address field in your request to that server.
You can get away with that kind of IP spoofing with UDP because UDP is a “connectionless” protocol. There’s no “handshake” when a request is made. The request arrives at the server, and the server just sends its response to the source IP of the request packet.
But HTTP uses a different network protocol – TCP. With the TCP protocol, there’s a three-way handshake at the networking level between the client and server, before the client can even send its request to the server. So if you put the victim’s IP address as the “source IP” address of your request, the server will try to perform the handshake with the intended victim. But the victim will think “WTF? I didn’t try to open this connection”, and the handshake will fail. And since the handshake fails, the bad guy’s small request can never actually be sent to the server. And so the unwitting server will never send its large response to the victim.
See: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_establishment
Hey rc,
I usually skim past questions like this (immediately). But kudos to you for taking the time to answer it.
A.
Oh hey, that’s what you think about this. Interesting that the ransom demand is the payload.
And also- as others have noted regarding the targets of these attacks- interesting that a lot of them are- themselves- accessories to an awful lot of spamming activity on the internet.
Digital Ocean and Choopa in particular stick out in my mind.
Turf war, maybe?
Not so sure about that description of UDP vs TCP. Neither protocol required authentication. A common example of ‘authentication not required’, years ago we studied speedtest.net’s method for estimating bandwidth, and discovered they use something called TCP Ping, which actually measures the speed you achieve opening a TCP connection. No authentication required to do that. UDP is even lighter weight. Authentication is going to happen higher up the stack. In fact, I have always argued that UDP is better if TCP is not reliable enough at establishing a connection using the default TCP/IP stack connection process. This mean, if you use UDP, you will have to build the connection management at layer seven (in the ISO model).
I do have to admit, having been away from this, I am just learning they are using UDP. That seems particularly heinous to me as UDP is an inherently unprotected protocol, and is particularly designed for receiving unsolicited messages. Such messages include things like older NFS installations, SNMP, etc.
Grrr, I am having a hard time staying away from an attitude toward severe punishment what seems to be a mostly young man’s domain, which will also throw a net around the juvenile offenders as well.
Incidentally, I notice my mostly standard base installation of Ubuntu 17.04 Server does not come with ‘memcached’ install. Thank you dweebs, for protecting me from my ignorance.
Thanks Eileen x