On Tuesday evening, KrebsOnSecurity.com was the target of an extremely large and unusual distributed denial-of-service (DDoS) attack designed to knock the site offline. The attack did not succeed thanks to the hard work of the engineers at Akamai, the company that protects my site from such digital sieges. But according to Akamai, it was nearly double the size of the largest attack they’d seen previously, and was among the biggest assaults the Internet has ever witnessed.
The attack began around 8 p.m. ET on Sept. 20, and initial reports put it at approximately 665 Gigabits of traffic per second. Additional analysis on the attack traffic suggests the assault was closer to 620 Gbps in size, but in any case this is many orders of magnitude more traffic than is typically needed to knock most sites offline.
Martin McKeay, Akamai’s senior security advocate, said the largest attack the company had seen previously clocked in earlier this year at 363 Gbps. But he said there was a major difference between last night’s DDoS and the previous record holder: The 363 Gpbs attack is thought to have been generated by a botnet of compromised systems using well-known techniques allowing them to “amplify” a relatively small attack into a much larger one.
In contrast, the huge assault this week on my site appears to have been launched almost exclusively by a very large botnet of hacked devices.
The largest DDoS attacks on record tend to be the result of a tried-and-true method known as a DNS reflection attack. In such assaults, the perpetrators are able to leverage unmanaged DNS servers on the Web to create huge traffic floods.
Ideally, DNS servers only provide services to machines within a trusted domain. But DNS reflection attacks rely on consumer and business routers and other devices equipped with DNS servers that are (mis)configured to accept queries from anywhere on the Web. Attackers can send spoofed DNS queries to these so-called “open recursive” DNS servers, forging the request so that it appears to come from the target’s network. That way, when the DNS servers respond, they reply to the spoofed (target) address.
The bad guys also can amplify a reflective attack by crafting DNS queries so that the responses are much bigger than the requests. They do this by taking advantage of an extension to the DNS protocol that enables large DNS messages. For example, an attacker could compose a DNS request of less than 100 bytes, prompting a response that is 60-70 times as large. This “amplification” effect is especially pronounced if the perpetrators query dozens of DNS servers with these spoofed requests simultaneously.
But according to Akamai, none of the attack methods employed in Tuesday night’s assault on KrebsOnSecurity relied on amplification or reflection. Rather, many were garbage Web attack methods that require a legitimate connection between the attacking host and the target, including SYN, GET and POST floods.
That is, with the exception of one attack method: Preliminary analysis of the attack traffic suggests that perhaps the biggest chunk of the attack came in the form of traffic designed to look like it was generic routing encapsulation (GRE) data packets, a communication protocol used to establish a direct, point-to-point connection between network nodes. GRE lets two peers share data they wouldn’t be able to share over the public network itself.
“Seeing that much attack coming from GRE is really unusual,” Akamai’s McKeay said. “We’ve only started seeing that recently, but seeing it at this volume is very new.”
McKeay explained that the source of GRE traffic can’t be spoofed or faked the same way DDoS attackers can spoof DNS traffic. Nor can junk Web-based DDoS attacks like those mentioned above. That suggests the attackers behind this record assault launched it from quite a large collection of hacked systems — possibly hundreds of thousands of systems.
“Someone has a botnet with capabilities we haven’t seen before,” McKeay said. “We looked at the traffic coming from the attacking systems, and they weren’t just from one region of the world or from a small subset of networks — they were everywhere.”
There are some indications that this attack was launched with the help of a botnet that has enslaved a large number of hacked so-called “Internet of Things,” (IoT) devices — routers, IP cameras and digital video recorders (DVRs) that are exposed to the Internet and protected with weak or hard-coded passwords.
As noted in a recent report from Flashpoint and Level 3 Threat Research Labs, the threat from IoT-based botnets is powered by malware that goes by many names, including “Lizkebab,” “BASHLITE,” “Torlus” and “gafgyt.” According to that report, the source code for this malware was leaked in early 2015 and has been spun off into more than a dozen variants.
“Each botnet spreads to new hosts by scanning for vulnerable devices in order to install the malware,” the report notes. “Two primary models for scanning exist. The first instructs bots to port scan for telnet servers and attempts to brute force the username and password to gain access to the device.”
Their analysis continues:
“The other model, which is becoming increasingly common, uses external scanners to find and harvest new bots, in some cases scanning from the [botnet control] servers themselves. The latter model adds a wide variety of infection methods, including brute forcing login credentials on SSH servers and exploiting known security weaknesses in other services.”
I’ll address some of the challenges of minimizing the threat from large-scale DDoS attacks in a future post. But for now it seems likely that we can expect such monster attacks to soon become the new norm.
Many readers have been asking whether this attack was in retaliation for my recent series on the takedown of the DDoS-for-hire service vDOS, which coincided with the arrests of two young men named in my original report as founders of the service.
I can’t say for sure, but it seems likely related: Some of the POST request attacks that came in last night as part of this 620 Gbps attack included the string “freeapplej4ck,” a reference to the nickname used by one of the vDOS co-owners.
Update Sept. 22, 8:33 a.m. ET: Corrected the maximum previous DDoS seen by Akamai. It was 363, not 336 as stated earlier.
LOL, what I get out of this attack, is kids crying that they (or their friends) got caught. And crying very loudly (620Gps worth).
Kids if you don’t want to get spanked, don’t be a dick. It’s really that simple. It’s not like you don’t know better. Just stop being a bad person.
I totally concur! Sounds like someone is pissed, and now they are flinging their boogers at the site. And hopefully they get caught
Someone must have a really bad sinus infection.
Yeah. I just compared this elsewhere to an online childish temper tantrum…
You, my friend, nailed it right on the head!
Wow! You certainly pushed the right button with your reporting, Brian. Keep up the good work. I wonder how long the bad guys can do this without revealing themselves somewhere along the line.
Regards,
On my home network, my Netgear router has no usage monitoring. It should. I would have to run a network sniffer in order to detect a bot on my network.
ISPs should have better tools. If ISPs can afford to ignore bots on their network, then ISPs have too much bandwidth. ISP usage monitoring should make them aware that an internal IP address is spewing garbage on their network.
And what made you think any significant portion of this attack came from an ISP?
Your statement is “not even wrong”.
I get what vb is saying.
These are bots, lots of them; it’s a pretty safe bet that a lot of the bots are at homes, which means that ISPs specify, purchase, and provide the network equipment at those premises. If that network equipment contained the ability to measure network utilization…something simple like mrtg…then it would be pretty simple for people to detect when they have a DDoS bot active on their home network. Without that, it’s a lot more challenging.
vb wasn’t saying that ISPs were the attackers…he (she?) was wishing that ISPs would provide people with the tools they need to easily detect when they are being used in an attack.
Actually, the intel I’m gathering suggests it’s not routers at issue, but mostly DVRs and some IP cameras. But when it comes to ISPs and hosters failing to filter spoofed traffic leaving your network, then I agree that’s a big problem for the ISPs.
He didn’t say it *came* from an ISP, read his comment again. He said ISPs should provide their customers better network monitoring tools so they can detect when one of their IoT devices has been botted.
ISP’s might have a vested interest in protecting their own equipment (the modem/router) and their network (the mainline infrastructure). But think about what your asking.
Most people would not use such tools even if every ISP provided them and pushed them. No one actually cares to understand any of this stuff enough to be so motivated.
Those that do understand it will then see such actions by ISP’s as ‘privacy intrusions’.
Nevermind the fact that these tools are (and have been) available to everyone. The ability to monitor your own network in this way is NOT incredibly expensive. Your not going to pay tens of thousands of dollars for it. Much of it could even be had for free.
Besides….
Why would you want your ISP controlling your network in this way? Hasn’t it already been proven to the world that ISP provided routers are just as bad if not worse than what the average person buys off the shelf?
YOU bought all those IoT devices. They belong to you. Your TV is yours. It belongs to you. Your ISP is not responsible for your TV or your thermostat or your refrigerator or your toaster. That job is yours.
It would be nice to see ISPs equip their customers better – I’m lucky in having an excellent UK ISP which gives their customers access to a lot of controls (second by second traffic, latency and loss graphs, packet capture facilities at their edge router) – if you get hit with a 600 Gbps DDoS, though, you’re looking for a new ISP in a hurry: even they just can’t shield customers from that kind of sledgehammer for long. (They were used as a trial run for the attack which knocked Janet – the UK academic/government network – offline for a while late last year; rather than the “brute force” flood here, that was a much more ‘intelligent’ attack, aiming to overload router control planes rather than the – faster – switching planes.)
The idea of allowing a profit motivated entity such as an ISP to place equipment on my private LAN is patently scary.
Looks like you’re still being hit. I tried loading this story from a few different ISPs and it failed on one and took quite a while on the other.
While a lot of DDoS attacks, like the ones you investigated, are motivated by actors looking for money, revenge, etc., not all of them are. Some have political underpinnings that can’t be summarily dismissed — and you have, here. You may have caught some “bad guys”, but you haven’t taken responsibility for what came after — which is throwing them to a justice system that is incompetent, aggressive, and almost always unfair… to everyone, victims and perpetrators alike.
–
By disclosing their names publicly, and prompting arrests, you’ve thrown them into a system that is badly broken. Maybe you console yourself with the notion that “if you can’t do the time, don’t do the crime”… but that’s hypocritical in the extreme, since many other people who assume the title of ‘security researcher’ are currently serving prison sentences for doing things few would consider criminal acts, or at least were not motivated by personal gain. Your fame has insulated you from the fate of others doing the exact same work… don’t mistake that for some kind of endorsement of your conduct.
–
You’re being intellectually dishonest by ignoring the political tapestry that goes with the technology. But then, I’m hardly the first to say that about your reports. It’s a blind spot in the entire industry you work in; An over-focusing on the technology at the expense of social intelligence. Unfortunately, that bias will continue to hold back the industry as a whole, and you as a professional, for a long time to come yet.
–
You may consider yourself a “researcher”, but you are also a political activist. Exploring why you can’t admit this publicly, or possibly even personally, would be a true contribution to the field, and a discussion that needs to be had.
you could have simplified and just said “derp”
My, what a disjointed sense of justice you have. Maybe we should reward these guys and give them high-paying jobs, you think?
For the record, the FBI and other investigative agencies were almost certainly tracking these turkeys long before I was. Nothing like that gets spun up overnight, unless maybe it’s a terror investigation or amber alert.
In any case, I’m not the one who put them in the justice system. They did, by their unbridled arrogance, cowardice, and avarice. To say nothing of their laziness. They practically put up a billboard about who and where they were and dared the world to chastise them like the spoiled children they are. And guess what?
The best journalists develop and publish stories that affect change and greater awareness where it is needed. If you want to call that activism, suit yourself. But the DDoS-for-hire industry needs a lot more sunshine, and I have no intention of pulling back on my reporting there.
Amen brother, keep up the good work!
Your site is one of my top 5 for tech info. Many articles have been a real eye opener. As it turns out shining a light makes you a target for the sick freaks. Keep on.
Are there any online scanners that can detect whether your IP address has open recursive DNS servers? I’m thinking something like the GRC Shields Up scanner.
There are. See:
http://openresolverproject.org/
Keep up the good work, Brian!
Although not quite the same thing, CAIDA is now publishing some very useful information that could give you some sense of whether your ISP allows IP spoofing. This is a voluntary effort, so the biggest offenders are not listed here, but it’s a start. More on this in a coming post.
https://spoofer.caida.org/recent_tests.php
Thank you. I had found that site myself earlier but was confused by the interface. I was thinking that you’re supposed to do a Check my IP Space, but my search results would just show a table with headers but no data.
However, lower down on that page there’s a link to “Test your IP Now!” that takes you here: http://www.thinkbroadband.com/tools/dnscheck.html
That seems to be a friendlier tool for checking for open DNS resolvers.
ELI5: So let’s say an attack succeeds in knocking the site off-line. So what? How long until it’s back on-line? How much damage has been done?
Does unleashing a huge attack also increase the chance of identifying the attacker? The bigger the attack, the bigger the chance of being identified?
What are the laws against this? Can people who buy attacks and its providers be arrested or sued?
To a site like this, the impact and costs are pretty minimal, taking this down is just them dick-waving. But for any business that trades online, the threat of some hours offline has some cost, but even that is mostly posturing because the cost to the attacker is trivial compared to matching DDoS protection. They could just as well keep the site down for a day, a week, a year, and the business owners know this. In this instance, obviously Brian upset some people with his investigations and reporting, and this result is not that surprising – but this is not an action that will stop him writing, nor stop people being able to read what he’s written.
If it can be proved a perpetrator is responsible for a DoS, and they’re in a judiciary that cares, then their actions have huge civil liabilities, as it’s wilfully causing financial damage (directly or reputational). Whether there is specific law to criminally prosecute them for the specifics of computer misuse or similar depends on the countries involved.
Fundamentally, these DDOS things are going to get *much* harder to mitigate before they get easier.
I *cannot* trust my PC, let alone my phone, IOT toy or broadband router/wireless access point not to get hacked. Each of these can “browse” (or direct otherwise normal traffic) at my favorite IP address. Whoever pwned me doesn’t care if I get blocked, I’m one of millions!
I can’t even tell if my own network traffic is legit or not…what with my stuff all phoning home to companies that get lots of universal love, like Microsoft and Apple.
So, until I can actually trust my computers…its going to get worse!
Brian, you must be doing something right. Thank you.
Oh, record DDoS? So what is this then – https://twitter.com/olesovhcom/status/778830571677978624 😀 1100 + 901 Gb/s DDoS.
If you read the article, you’ll see that the record is in relation to DDoS observed by Akamai/Prolexic.
“Martin McKeay, Akamai’s senior security advocate, said the largest attack the company had seen previously clocked in earlier this year at 363 Gbps. But he said there was a major difference between last night’s DDoS and the previous record holder”
At least things are going to get much worse from here. So we have that to look forward to.
Welcome back, Brian. I look forward to you exposing these latest scumbags to the world. And a big congrats to whoever decided to take on the burden of hosting you. Perhaps you should start a GoFundMe campaign (or something similar) to help pay some of the expenses for keeping you up, and fighting the good fight.
Good to see ya back.
Glad to see that the site is back. Will be checking for the description of what’s happened and happening. Good work.
Thanks for all the work and news you provide, Brian. Welcome back and I hope you’re able to get help with this, and stop these cowards. I have read the site daily for years.
Have a lovely afternoon, Brian.
Thanks for getting back online so quickly. No thanks to Akamai for giving up on you.
I hope your alliance with Google can help keep you online and track down the miscreants who attack the net.
No thanks to Akamai? They provided free security service to the site for ages! The protection cost a Akamai a great deal of cash and labor, and I am certain Akamai would love to add Krebs as a paying customer!
To blame Akamai for recinding a free service is just plain foolish.
Mike sr.
If enough of your readers who subscribe to your updates and follow your blog were to send you five dollars through PayPal ( or even send a five dollar bill to you in the mail ), I really believe we would be standing up for the rights of free speech and an open press in America
This site had earlier a PayPal-button, but not anymore. Why?
Welcome back!
I look forward to reading your future articles about the criminal convictions for the clowns who did this.
I hope the clowns discover that it really wasn’t worth the effort to end up with some long stretch in Club Fed. If you’re going down, at least make it a big one. Go e-rob a bank and steal a billion dollars, or do something else where at least the payoff is worth the potential of a long prison sentence.
Very glad to see you back!!!! : ) Go Krebs Go
Ppeople who “research” on the internet have become vigilantes who scream “Free speech!” whenever they get punched in the face for doing law enforcement’s job.
–
Kreb’s… kindly shut up about the free speech angle. You were a vigilante. Don’t cloak yourself in higher morals to obscure the truth of it: Just own it.
Sometimes you read a comment so disconnected from reality, so void of any understanding of the issues at hand and so hostile, you have to wonder how the commenter even makes it through a day without going super saiyan and self destructing on the spot.
I can just imagine the red glow emanating from your pulsing angry head over subjects you literally have zero understanding of.
Prior to the ready availability of information provided by the internet the job of a journalist involved a lot of footwork. Even if first given information by a source, the onus was on the reporter to verify, pursue, and develop a more complete picture.
Investigative journalism is seen as unnecessary by some in the modern era of information glut and panel-style opinion pieces repeating existing news from multiple angles. However the best journalists are still those who do the work themselves and bring to light information that wouldn’t otherwise be public. That the the reporting on that subject corresponds with (or even triggers) law enforcement involvement is not vigilantism, it is the natural consequence of good journalism casting light on illegal activities.
If Brian Krebs is, in fact, batman, and the criminals in question were found by the police already trussed up with a copy of Brian’s report on their laps, then perhaps the vigilante accusation would work. But batman ‘historically’ only does the trussing up, and makes others do the hard work of actually gathering evidence and doing the reporting.
It sounds like this is also a precursor to a state-sponsored preemptive attack, with its final goal to be online voting this November. #TrialRun
looks like they are back. You cannot hit the front page. 403…
Wow, you sure have some sworn enemies to solicit such a big DDoS attack. Keep telling the truth man!
Well : ) could u explain what botnet or what was used to carry out this attack? i mean i could find a RCE and then add a function to my botnet to spread by exploiting the infected victims : )
LOL @ prolexic and akamai.
Have a look at these attacks on the same day, much bigger. I think it’s bad marketing for prolexic/akamai. To put it simply they couldn’t handle the DDoS and it seems the story about cost is more about covering this fact up.
http://www.scmagazineuk.com/ovh-suffers-11tbps-ddos-attack/article/524826/
https://isc.sans.edu/forums/diary/The+era+of+big+DDOS/21511/
https://twitter.com/olesovhcom/
Hahaa: “Over the past week, the company has been subjected to an attack greater than the one suffered by Krebs on Security.”
It looks like there is a race going on: “Who’s being hit the worst”.
IDEA FOR THE FUTURE:
Trusted agencies should report to local internet providers, that one of his IP-address is part of an attack. The internet provider than have to cut the internet access and the only site the user should see is something like “your computer (devices) have been compromised, please take action to clean them”. The user shoukld be able to access internet maybe just for a couple of hours a day to load software and contact technical support. After some days he should get back full internet access.
Well, I know it’s not easy, but the users have to be informed and forced to take action!
Is it possible to get a visualization of the DDOS attack? Like graphics of some sort showing the amount of data pushed?
i think that you deserve it, you named the guys, in real world you’ve had broken legs.
I believe that to DDOS a blog you should be really not having what you do…
Most likely somebody just wanted to test something and from what has happened the test looks quite successful..
Your blog was down, Akmai is taking money from their paying customers for nothing…
If you want real protection still RTBH + a solid ISP with well developed peering rules.
RFC 5635 – Remote Triggered Black Hole Filtering with Unicast … that only BGP …
https://tools.ietf.org/html/rfc5635