On Tuesday evening, KrebsOnSecurity.com was the target of an extremely large and unusual distributed denial-of-service (DDoS) attack designed to knock the site offline. The attack did not succeed thanks to the hard work of the engineers at Akamai, the company that protects my site from such digital sieges. But according to Akamai, it was nearly double the size of the largest attack they’d seen previously, and was among the biggest assaults the Internet has ever witnessed.
The attack began around 8 p.m. ET on Sept. 20, and initial reports put it at approximately 665 Gigabits of traffic per second. Additional analysis on the attack traffic suggests the assault was closer to 620 Gbps in size, but in any case this is many orders of magnitude more traffic than is typically needed to knock most sites offline.
Martin McKeay, Akamai’s senior security advocate, said the largest attack the company had seen previously clocked in earlier this year at 363 Gbps. But he said there was a major difference between last night’s DDoS and the previous record holder: The 363 Gpbs attack is thought to have been generated by a botnet of compromised systems using well-known techniques allowing them to “amplify” a relatively small attack into a much larger one.
In contrast, the huge assault this week on my site appears to have been launched almost exclusively by a very large botnet of hacked devices.
The largest DDoS attacks on record tend to be the result of a tried-and-true method known as a DNS reflection attack. In such assaults, the perpetrators are able to leverage unmanaged DNS servers on the Web to create huge traffic floods.
Ideally, DNS servers only provide services to machines within a trusted domain. But DNS reflection attacks rely on consumer and business routers and other devices equipped with DNS servers that are (mis)configured to accept queries from anywhere on the Web. Attackers can send spoofed DNS queries to these so-called “open recursive” DNS servers, forging the request so that it appears to come from the target’s network. That way, when the DNS servers respond, they reply to the spoofed (target) address.
The bad guys also can amplify a reflective attack by crafting DNS queries so that the responses are much bigger than the requests. They do this by taking advantage of an extension to the DNS protocol that enables large DNS messages. For example, an attacker could compose a DNS request of less than 100 bytes, prompting a response that is 60-70 times as large. This “amplification” effect is especially pronounced if the perpetrators query dozens of DNS servers with these spoofed requests simultaneously.
But according to Akamai, none of the attack methods employed in Tuesday night’s assault on KrebsOnSecurity relied on amplification or reflection. Rather, many were garbage Web attack methods that require a legitimate connection between the attacking host and the target, including SYN, GET and POST floods.
That is, with the exception of one attack method: Preliminary analysis of the attack traffic suggests that perhaps the biggest chunk of the attack came in the form of traffic designed to look like it was generic routing encapsulation (GRE) data packets, a communication protocol used to establish a direct, point-to-point connection between network nodes. GRE lets two peers share data they wouldn’t be able to share over the public network itself.
“Seeing that much attack coming from GRE is really unusual,” Akamai’s McKeay said. “We’ve only started seeing that recently, but seeing it at this volume is very new.”
McKeay explained that the source of GRE traffic can’t be spoofed or faked the same way DDoS attackers can spoof DNS traffic. Nor can junk Web-based DDoS attacks like those mentioned above. That suggests the attackers behind this record assault launched it from quite a large collection of hacked systems — possibly hundreds of thousands of systems.
“Someone has a botnet with capabilities we haven’t seen before,” McKeay said. “We looked at the traffic coming from the attacking systems, and they weren’t just from one region of the world or from a small subset of networks — they were everywhere.”
There are some indications that this attack was launched with the help of a botnet that has enslaved a large number of hacked so-called “Internet of Things,” (IoT) devices — routers, IP cameras and digital video recorders (DVRs) that are exposed to the Internet and protected with weak or hard-coded passwords.
As noted in a recent report from Flashpoint and Level 3 Threat Research Labs, the threat from IoT-based botnets is powered by malware that goes by many names, including “Lizkebab,” “BASHLITE,” “Torlus” and “gafgyt.” According to that report, the source code for this malware was leaked in early 2015 and has been spun off into more than a dozen variants.
“Each botnet spreads to new hosts by scanning for vulnerable devices in order to install the malware,” the report notes. “Two primary models for scanning exist. The first instructs bots to port scan for telnet servers and attempts to brute force the username and password to gain access to the device.”
Their analysis continues:
“The other model, which is becoming increasingly common, uses external scanners to find and harvest new bots, in some cases scanning from the [botnet control] servers themselves. The latter model adds a wide variety of infection methods, including brute forcing login credentials on SSH servers and exploiting known security weaknesses in other services.”
I’ll address some of the challenges of minimizing the threat from large-scale DDoS attacks in a future post. But for now it seems likely that we can expect such monster attacks to soon become the new norm.
Many readers have been asking whether this attack was in retaliation for my recent series on the takedown of the DDoS-for-hire service vDOS, which coincided with the arrests of two young men named in my original report as founders of the service.
I can’t say for sure, but it seems likely related: Some of the POST request attacks that came in last night as part of this 620 Gbps attack included the string “freeapplej4ck,” a reference to the nickname used by one of the vDOS co-owners.
Update Sept. 22, 8:33 a.m. ET: Corrected the maximum previous DDoS seen by Akamai. It was 363, not 336 as stated earlier.
They say the traffic seems to come from “everywhere”, but the botnet owners typically don’t want to soil their own nest as it were. So I would be interested if there is a region of the world where you have no traffic, but where one would reasonably expect to see it.
You now have the biggest honeypot log ever!
Hope someone puts that Intel to good use.
Looking forward to more on the incident.
Fun fact, our friend chippy1337 from BackConnect released LizKebab, without a doubt kick-starting the IoT botnet scene into high gear. Torlus, etc are all based on the same codebase.
https://github.com/gh0std4ncer/lizkebab
GRE can be spoofed in a sense, according to Cisco docs its exempt from uRPF/BCP38 among other tunnel protocols.
Correction, Can be spoofed, but the real IP will be in the inner IP header of the encapsulated packet.
That’s not true, the inner IP address is not validated by any traditional routers. Both the encapsulated IP header and inner IP header may be spoofed. Please do not spread misinformation
AWESOME?
That’s probably the first time in many years I’ve seen or heard that word properly used, Bill. So thank you for that.
Might be time to buy stock in Akamai 🙂 Certainly a feather in their cap.
Expect marketing calls about alternative DDoS protection services… or a BGP hijack. I always forget and get the two confused.
Wow, you must really be disliked, Brian. I’m thinking about all the lost revenue the attackers must be passing up on to try and bring your site down…!
It amazes me that providers can turn a blind eye to the abuse that is being perpetrated via their systems. To me, that is one of the biggest problems: a non self-policing Internet. Those that don’t control and take responsibility for what happens on their systems should be blocked and disconnected, IMO.
We currently have a 10-mile high wall around our country compared to the zero barrier access to our networks and systems. System admins know how bad things are, but we aren’t going to do anything until we experience a 9/11 scale event in cyberspace.
And while I’m at it I just want to say that the current telecommunications attack situation is just as bad. Yesterday I had another one of those calls from “Windows support desk” who wanted to fix my computer. I keep them on the line until they start cursing me and every swear word is music to my ears. Today I got one call about an IRS arrest warrant that was going to be served on me if I didn’t pay $6,780 in back taxes, interest, and penalties from a 2013 tax return I filed with serious errors. Then later in the day I got to grant awards, one for $5k and one for $9k. Maybe I get a lot of these because I am 64, but my god I feel bad for people that are taken in my these #(#&$% criminals. How much longer are we going to emulate a fluffy barn yard animal…? Baa? Bah!
I’m based in the UK and currently taking my internet service from British Telecom, the largest UK phone provider. I used to get 4 or 5 “Windows Technical Support” scams each week, until, in March this year, one caller actually quoted me, correctly, an account number that is only printed on the paper copy of my phone bill (and nowhere else).
That one, single, tiny slip gave them away. This wasn’t some random boiler-room criminal gang. No. It was something a lot smarter than that. It was a criminal gang that had set up operations *inside* my telco provider’s own call center. They were using my telco’s own phone system to call me in the UK (which was how they were paying for and masking the calls) and this was one of the reasons that I could not use the last-caller-number dial-back to figure out who they were.
I called the police and eventually, after much hassle, got them to give me a crime number.
Then I called my phone company and told them what had happened. At first they seemed disinterested, but when I told them I had a crime number, they suddenly paid a bit more attention. I then suggested that all the telco needed to do was look at the three or four people I had been speaking with on a recent issue, because I was sure it was related to the scam calls. The calls stopped, overnight.
I went from an average of one per day to zero.
The telco is denying everything (I.e. Not admitting they had a problem with rogue employees) newly although this is something I would find very difficult to provide actual proof for, all the anecdotal evidence I have suggests that my “Windiws Technical Support” calls were being made by rogue employees of my telco.
I reckon they had set up a boiler room scam by working a few extra hours per day…
This won’t be the case every time, but it’s always worth seeing what you can get them to disclose…
Huh. Interesting!
I usually don’t pick up calls from numbers I don’t recognize nowadays. If no one leaves a voicemail message, I’ll look up the number that appeared on Caller ID, and see if it’s a legitimate call or not.
A 9/11/2001 cyber attack? And what exactly would that look like? Google offline for ten minutes? The OPM hack? Somewhere in the middle?
Did you not see Die Hard 28? Think critical infrastructure like water, power, stocks, traffic lights rather than your ability to search for porn or apply for a job.
WOW!?
Damn cyber criminals
Was the site (krebsonsecurity) knocked down by this attack? Was this the biggest attack EVER, on anyone? Were other web sites affected by the attack? Was this attack actually bigger than the ones you hear about directed at big name companies? It seems that the attack amounted to a dynamite attack against an ant hill. How do I find out for certain if my computer is part of a botnet? Sorry for the questions but I’m just a civilian trying to get a sense of scale, a feel for how much harm these things cause.
Was the site (krebsonsecurity) knocked down by this attack?
No
Was this the biggest attack EVER, on anyone?
Maybe. Lots of places get attacked with huge attacks, but few of them acknowledge it. The largest DDoS seen by Arbor, probably Akamai’s largest competitor, was 560 Gbps.
Were other web sites affected by the attack?
According to Akamai, no.
Was this attack actually bigger than the ones you hear about directed at big name companies?
Yes
How do I find out for certain if my computer is part of a botnet?
This is a much harder question to answer.
Congrats, these guys must *really* hate you.
Goes to show you’re doing a good job!
Is my pc part of a botnet?
Are my light bulbs? It seems to me that soho network equipment should begin implementing default outgoing firewalling as people start using these cheap wifi-enabled devices.
https://mjg59.dreamwidth.org/40397.html
Brian, I beg to differ with you and Akamai. At least from Comcast (Atlanta), your site was indeed offline when I tried to connect a couple of times yesterday. At some times, html/css/(whatever) would load, but not any of the images, and at other times, the images would load but not much else, and the first time it just timed out without anything loading (i checked view-source to see because I was expecting it to say like ‘hacked by freeze’)
Greg,
How do I know if my computer is part of a botnet?
Generally speaking, one way is: http://www.abuseat.org, then “lookup”.
Hmmmm Another link offered for my clickage. And I almost clicked on it. Too bad I can’t trust anyone
Why can’t you find a good backhoe when you need one?
Brian,
We, of the right side of the tracks, appreciate your work. We also realize that your job entails the risk of much more than just DDoS attacks.
Thank You
All you had to do was hijack the C&C to stop the attack Krebs!
It’s good to see the lulz being replaced by the failz.
665 was really huge, and this one is the biggest DDos attack that ever seen. Still your site is up… Amazing
Someone hit it offline https://twitter.com/BannedOffline/status/778780149957423105
10 hours ago I was not able to connect to this site for hours and gave up. No wonder it was timing out. 600 gbps is massive and I believe the last series of articles were the reason. First the bust in Israel and now the last one about the pecker you can’t trust on.
Looks like you hit a sore spot Brian, congrats!
I would not be surprised if DDOS “protection services” employed the use of “services” like vDOS to amplify their “business model”.
I hope you’re not part of Akamai’s normal DDOS plans that charge you for the size of the attack. Can’t imagine what that bill looks like if so.
Akamai includes fee protection on their KSD product that would eliminate any overages from the DDoS traffic.
In a parallel universe, the public became enraged by the practice of companies hiring the cheapest developers they could find and then ignoring all sane engineering practices years ago. In that universe, executives for companies that ignore the warnings of their software engineers and let people with business degrees decide when a product is ‘secure enough’ are prosecuted for criminal negligence. In that universe, these sorts of things aren’t expected as the new normal.
We don’t live in that universe.
That would be great! Only a fantasy! Sigh
“We don’t live in that [legal] parallel universe” — yet.
Cf. A History of American Law by Lawrence M. Friedman. Or consider the example of the Massachusetts legislature making it illegal for anyone to enforce the Fugitive Slave Act, passed by the United States Congress. The laws must change.
Congratulations on weathering the attack, and congratulations on doing work that provoked it.
A security architect at Alibaba told me in Oct 2015, they are seeing DDoS of several hundred GBit within the Alibaba cloud on a daily base. He said they were getting used to fighting 500 GBit. And that was a year ago.
Contact available on request.
Isn’t it time to make some fundamental changes? All those IP’s, should be sent to corresoponding hosting companies to inform owner of IP to clean system. If no cleaning happens, then block user. IF ISP doesn’t cooperate, blocking ISP is next step. YES… I know this is a huge work, but we don’t have better solution. Owners of computers/laptops/IoT should understand, that having device isn’t just consuption of entertainment but also it means awarness and responsibility of taking care of systems.
Aargh!
I’ve been advocating that since 1993. I even put together a business plan to create a company to do just that…but the liability insurance and performance bonds were WELL BEYOND anything I could afford (or even finance.)
same same. excellent, and keep up the good work! I salute you.
If this was an IoT based botnet then things are only going to get worse. With projections of billions of IoT devices being online and with no sign of the manufacturers being interested in security the broader economic impacts of these sort of attacks can hardly be imagined. Attacking multiple targets at once will probably render the DDoS protection services impotent as they have several clients all clamouring for bandwidth, and if the targets are financial institutions then everything could grind to a halt.
re: “With projections of billions of IoT devices being online and with no sign of the manufacturers being interested in security the broader economic impacts of these sort of attacks can hardly be imagined.”
That is why I don’t have and will not have, in the foreseeable future, any IoT devices. I had a new home built a few years ago. It came with a Wi-Fi enabled thermostat. Nope, it will never be connected to the internet.
Size does not matter 😉
Except for vendors may be…
Was it just after you posted the last article? Kind of a big coincidence
Hello,
Im thinking on that but,
Do you know whats the purpose of use gre packets?
When they use dns is because the can reflect and amplify the original request, but why use gre? what is the advantage?
thanks.
How does something like a baby monitor or Nest thermostat contribute to this kind of thing?
cameras, nest thermos, ovens, etc., any iot device has an os (like android) and is connected to the internet and can be hacked. most people hook stuff up and then don’t do anything to secure that config. ddos will get worse.
To be correct, an order of magnitude is putting another zero behind the number – so 336 giga bytes becomes 3.360 giga bytes – much larger that 620 giga bytes. So the attack was not “several orders of magnitude greater” – it was merely greater.
Don’t get me going about exponential growth.
that should be 3.360 tera bytes
Or more likely a comma.
Many countries (in Europe mainly, I believe) use a period as a thousands separator and a comma as a decimal point. So “3.360 giga bytes” would have been correct; “three thousand three hundred giga bytes” is indeed an order of magnitude more than 336 gigs.
@Santa – the article said orders of magnitude more than needed to knock a site offline, not orders of magnitude more than the previous attack. FYI, it only takes a few Gbps DDoS to know your average site offline.
so a typical site gets knocked off with, say 3 mb of traffic? (i. e., “orders of of magnitude” as in plural). cite your sources on this fascinating topic.
I think you need to read that paragraph again, then you’ll have that ohh moment.
i merely asked him to cite sources
3MB is 1/1000 the size of 3Gbps (which can knock a website offline).
The attack being referred to was more than 200 times 3Gbps. That is definitely “orders of magnitude.”
Glad they were able to keep you online. I’m sensing an article or two out of this. “Is my computer a Bot”