Criminals this morning massively attacked Dyn, a company that provides core Internet services for Twitter, SoundCloud, Spotify, Reddit and a host of other sites, causing outages and slowness for many of Dyn’s customers.
Twitter is experiencing problems, as seen through the social media platform Hootsuite.
In a statement, Dyn said that this morning, October 21, Dyn received a global distributed denial of service (DDoS) attack on its DNS infrastructure on the east coast starting at around 7:10 a.m. ET (11:10 UTC).
“DNS traffic resolved from east coast name server locations are experiencing a service interruption during this time. Updates will be posted as information becomes available,” the company wrote.
DYN encouraged customers with concerns to check the company’s status page for updates and to reach out to its technical support team.
A DDoS is when crooks use a large number of hacked or ill-configured systems to flood a target site with so much junk traffic that it can no longer serve legitimate visitors.
DNS refers to Domain Name System services. DNS is an essential component of all Web sites, responsible for translating human-friendly Web site names like “example.com” into numeric, machine-readable Internet addresses. Anytime you send an e-mail or browse a Web site, your machine is sending a DNS look-up request to your Internet service provider to help route the traffic.
ANALYSIS
The attack on DYN comes just hours after DYN researcher Doug Madory presented a talk on DDoS attacks in Dallas, Texas at a meeting of the North American Network Operators Group (NANOG). Madory’s talk — available here on Youtube.com — delved deeper into research that he and I teamed up on to produce the data behind the story DDoS Mitigation Firm Has History of Hijacks.
That story (as well as one published earlier this week, Spreading the DDoS Disease and Selling the Cure) examined the sometimes blurry lines between certain DDoS mitigation firms and the cybercriminals apparently involved in launching some of the largest DDoS attacks the Internet has ever seen. Indeed, the record 620 Gbps DDoS against KrebsOnSecurity.com came just hours after I published the story on which Madory and I collaborated.
The record-sized attack that hit my site last month was quickly superseded by a DDoS against OVH, a French hosting firm that reported being targeted by a DDoS that was roughly twice the size of the assault on KrebsOnSecurity. As I noted in The Democratization of Censorship — the first story published after bringing my site back up under the protection of Google’s Project Shield — DDoS mitigation firms simply did not count on the size of these attacks increasing so quickly overnight, and are now scrambling to secure far greater capacity to handle much larger attacks concurrently.
The size of these DDoS attacks has increased so much lately thanks largely to the broad availability of tools for compromising and leveraging the collective firepower of so-called Internet of Things devices — poorly secured Internet-based security cameras, digital video recorders (DVRs) and Internet routers. Last month, a hacker by the name of Anna_Senpai released the source code for Mirai, a crime machine that enslaves IoT devices for use in large DDoS attacks. The 620 Gbps attack that hit my site last month was launched by a botnet built on Mirai, for example.
Interestingly, someone is now targeting infrastructure providers with extortion attacks and invoking the name Anna_senpai. According to a discussion thread started Wednesday on Web Hosting Talk, criminals are now invoking the Mirai author’s nickname in a bid to extort Bitcoins from targeted hosting providers.
“If you will not pay in time, DDoS attack will start, your web-services will
go down permanently. After that, price to stop will be increased to 5 BTC
with further increment of 5 BTC for every day of attack.NOTE, i?m not joking.
My attack are extremely powerful now – now average 700-800Gbps, sometimes over 1 Tbps per second. It will pass any remote protections, no current protection systems can help.”
Let me be clear: I have no data to indicate that the attack on Dyn is related to extortion, to Mirai or to any of the companies or individuals Madory referenced in his talk this week in Dallas. But Dyn is known for publishing detailed writeups on outages at other major Internet service providers. Here’s hoping the company does not deviate from that practice and soon publishes a postmortem on its own attack.
Update, 3:50 p.m. ET: Security firm Flashpoint is now reporting that they have seen indications that a Mirai-based botnet is indeed involved in the attack on Dyn today. Separately, I have heard from a trusted source who’s been tracking this activity and saw chatter in the cybercrime underground yesterday discussing a plan to attack Dyn.
Update, 10:22 a.m. ET: Dyn’s status page reports that all services are back to normal as of 13:20 UTC (9:20 a.m. ET). Fixed the link to Doug Madory’s talk on Youtube, to remove the URL shortener (which isn’t working because of this attack).
Update, 1:01 p.m. ET: Looks like the attacks on Dyn have resumed and this event is ongoing. This, from the Dyn status page:
This DDoS attack may also be impacting Dyn Managed DNS advanced services with possible delays in monitoring. Our Engineers are continuing to work on mitigating this issue.
Oct 21, 16:48 UTCAs of 15:52 UTC, we have begun monitoring and mitigating a DDoS attack against our Dyn Managed DNS infrastructure. Our Engineers are continuing to work on mitigating this issue.
Oct 21, 16:06 UTC
Luckily, this attack was relatively short. It will probably take an attack lasting several hours or even days before the internet community starts developing a serious plan to deal with the unsecured IoT devices powering these attacks.
It seems to me that every time someone ‘exposes’ these crooks– that they are then targeted BY those same. Including you– Brian. We need to keep the pressure on the crooks– and pressure the owners of weak websites/equipment to harden themselves against the attacks in the 1st place.
Plain old common sense. Too bad common sense is so seemingly UNcommon.
This website was a bit slow on loading today, hopefully you are not under attack.
Was the attack effectively shunted/mitigated or did the attackers call it off? Proof of concept to test their capabilities and just send a message?
Maybe I’m getting to be an old and bitter infosec guy (no maybe about), but this trend of outsourcing everything possible is gonna come back to bite us all. I’ve been fighting the movement of my organization’s data to “THE CLOUD!” and the outsourcing of essential infrastructure to Whatever-aaS and I’ve had so many I Told Ya So dances lately I’m sore.
“I told you so dance” sounds like the absolute best practice in terms of reacting to an attack on your company’s resources. Maybe they’ll listen to you more if you can do more for them than bark warnings and say “I told you so” when the warnings come true.
unfortunately most corporate budgets only approve these things after the “i told you so” moment.
you can warn and warn and warn but until the castle is attacked everyone thinks you are full of bull and will not approve budgets for security projects.
Investment requires estimates on return. No reason to spend money you can’t justify. Maybe get better at your job?
That is to say, if you wait for an attack to happen to provide a useful threat model and cost estimate of each level of compromise, then you are the problem with your company’s IT, not the bean counters.
Some resources are best implemented by a handful of experts and used as pooled shared resources. Any typical company running its own DNS service couldn’t possibly withstand a similar attack. And if each company independently locally provisioned DNS, they’d probably all be under-provisioned relative to Dyn, leaving them all collectively vulnerable to a distributed variation of this attack.
The mistake companies made was not outsourcing, but sole-sourcing.
Brian, what does it mean to say a DDOS attack consists of 600 or 800 Gbps? Does your connection through your ISP support that much bandwidth? Where does that volume occur? Where is the Gbps measured? Thanks,
Super question. Please, Brian: do tell…
His website was (at the time) protected by Akamai’s DDoS Protection Service; so the bandwidth was hitting them, not his actual web host. The size of the attack caused Akamai to drop Brian as a customer (albeit the service was being provided to him pro bono). He’s now being protected by Google’s Project Shield, which is specifically built to keep journalism sites from being blasted offline by DDoS. If someone cooks up something big enough to start bothering The G, then the entire Internet is screwed.
Its the amount of continuous traffic on the line at a central point. Say your ISP has a 200 Gigabit link (2 100 gig fiber connections bonded) and they have provided all of that bandwidth to your server. They can only support 200 gigabit of traffic before heavy queueing, latency and finally dropping packets starts happening. If a botnet has the ability of sending a collective amount of traffic continuously that exceeds 200 gigabit then a denial of service will begin to happen and your services as well as anything the ISP also hosts on that line will slow to zero basically. Its like drinking from a firehose, everything is good when its a straw then gets more saturated when its a water hose and then finally 99% of the water falls on the ground when you try to drink from a fire hose. Yes I am simplifying it here for your ease of understanding as well as using the theoretical link sizes but you should get the point now.
Computers can have viruses that help relay attack information. Your computer can be sending data to the attackers target without you having knowledge of it.
Well, that explains the 504’s I kept getting this morning trying to access WSJ
This is the new beginning of norm with IOT .
You do an excellent work and glad to see that you have not cowed down .
Well done and keep it up.
This explains how the world has to devise some way to fight the DDoS attacks. Smaller businesses are at total mercy of these extortionists. And even the big ones like Twitter feel the heat when even a DNS service it relies on gets DDoSed. So information security is a layered idea and every layer is important. Even those too which you do now own.
I wonder if, in the back ( or front) of Brian’s mind that he may be thinking, and patiently waiting for the next big bang DDoS attack on his domain to see how well he will fare under his new perch.
I wonder if the size and duration of the attack will be listed on the monitoring sites like this one:
http://www.digitalattackmap.com/#anim=1&color=0&country=ALL&list=0&time=16434&view=map
I wonder if this is a dark side to Net Neutrality? If service providers “billed” customers differently for IoT traffic than legitimate traffic there might be incentive to groom the traffic.
On the other hand, Net Neutrality encourages providers to remain agnostic and unaware of the damage their customers participate in causing. Its kind of like protecting Privacy or Anonymity encourages abuse of resources … to the detriment of everyone.
So…do you work directly for the NSA, or is your position a subcontract?
Really though, this should encourage devs to harden IoT device security to reduce exploits. Quit shipping out pieces of equipment that are built with shoddy, exploitable code, or are configured insecure out of the box.
In parallel, if people are going to be using the Internet, they need to educate themselves enough to minimize their threat vectors. The days of “my kid is a computer whiz and set-up all my wifi and banking account security, so I’m probably fine.” are over. The Internet isn’t going anywhere. People need to take responsibility for themselves and protect their IP Cameras, mobile devices, etc.
Reducing privacy is not the answer. We do not want the ISP policing and filtering our Internet traffic.
Lol.. very hyperbolic.. though I equally foudn the suggestion that devs would be incentivized to clean up the mess. Most are probably from an East Asian company and long gone.
The real devs that [could] clean up the mess would be the bot writers themselves. They are engaged in fighting over IoT devices innoculating as the co-opt them for their Borg Collectives.
Nation-states (NSA if you prefer) would do better to strategize by infecting IoT and innoculating them and then laying in wait.. like seeding a mine field both here and abroad.
A DDOS event could be use for good or bad after all, say a Nuclear Power plant were suddenly attacked from abroad.. hitting it with a DDOS blots out Command and Control events at the exactly worst time for an attacker.. you could look upon it as [switching off the internet] remotely without a court order under times of seige.
Same with a CryptoRansome-ware vent.. if detected in process.. you could halt the attack or slow it down to get the upper hand.
In a way I guess that is traffic shaping and Net Neutrality is just sticking your fingers in your ears yelling la-la-la-la and hoping for the best.
Even with Net Neutrality, ISPs could still filter outbound traffic to ensure that the egress IP address was not spoofed. That would be a big help so that DDos traffic could be more effectively filtered.
This isn’t really a consequence of net neutrality, it’s more a consequence of common carrier. ISPs bill by general usage and don’t inspect packets. Nor do they generally spend much time investigating individual customers. And because these networks are quite large, the number of customers an ISP would need to investigate could be fairly large…
Most entities aren’t designed to handle things at large scales. Investigating a single bandwidth leach, or a single alleged “pirate” is doable. But chasing down 50-1000 toasters and explaining the problem to the respective clients?
Also, keep in mind that while we see these devices as rogue, they probably aren’t purely evil, in most cases, they are probably still performing their intended function, and blocking them entirely would interfere with that and upset their paying customers.
Forget the sites the articles mention. I was having very serious troubles getting to the Guardian’s site, pictures not loading, and worse, I couldn’t even log onto my hosting provider for my email.
Now, I’m on Verizon FIOS, and my system (Linux, a eal o/s) couldn’t even ping hostmonster.com, it couldn’t find the name, until I manual added nameserver 8.8.8.8 (one of google’s) to my resolv.conf. Then it started working.
That tells me that it was overloading nameservers in a *LOT* of places.
mark
Forget the sites the articles mention. I was having very serious troubles getting to the Guardian’s site, pictures not loading, and worse, I couldn’t even log onto my hosting provider for my email.
Now, I’m on Verizon FIOS, and my system (Linux, a eal o/s) couldn’t even ping hostmonster.com, it couldn’t find the name, until I manual added nameserver 8.8.8.8 (one of google’s) to my resolv.conf. Then it started working.
That tells me that it was overloading nameservers in a *LOT* of places.
Oh, and that was close to 08:00 EDT. Now I’m wondering if there’s a political side to it – right now, 12:37 EDT, I cannot reach Paul Krugman’s blog in the NYT.
mark
As of 11:00-ish PST – the attacks appear to have started up again. From the outside it appears to be targeting Dyn, which then hits lots of their customers via failing DNS lookups.
https://news.ycombinator.com/item?id=12759520
Sites down:
– DYN
– Twitter
– Etsy
– Github
– soundcloud
– spotify
– heroku
– pagerduty
– shopify
– intercom
I am wondering about whether the manufacturers of IOT things in China are not incorporating any security only because it is cheaper, or because the chinese government would like to have easily converted spy/attack devices in most American/European homes.
Reddit was at the forefront of digging into the Wikileaks, and Twitter was a major distribution outlet for that data.
How do you stop that? DDOS yourself, or get someone to do it.
Claim its hackers.
Profit.
I think Dyn is lying, as we (and many of our customers) still can not resolve our host names.
Adam, Dyn seems to me to have been pretty transparent about this from the start. They’ve posted updates on their status page which is linked to in several places in this story. According to the latest update there, the attacks have resumed
Update – This DDoS attack may also be impacting Dyn Managed DNS advanced services with possible delays in monitoring. Our Engineers are continuing to work on mitigating this issue.
Oct 21, 16:48 UTC
Investigating – As of 15:52 UTC, we have begun monitoring and mitigating a DDoS attack against our Dyn Managed DNS infrastructure. Our Engineers are continuing to work on mitigating this issue.
Oct 21, 16:06 UTC
I do not think that providing protection would be an effective solution because no matter what organizations do, they will not be able to accommodate attack bandwidth.
May be thinking the other way around and stopping the attack from happening at first place would be an effective solution. Something like Shodan database of IoT devices IPs could be used to build an IP list that ISPs can use to block OUTBOUND connections from these source IPs -or applying some sort of restrictions based on ports- doing it gradually with one category after another. (i.e. starting with cameras, then move to home appliances, then move to a 3rd category…etc)
This just highlights a fundamental security problem: The web is far too centralized. Too many ways to knock out one service provider and bring down large swaths of the Internet.
What’s going on is certainly criminal, and needlessly disruptive to many peoples’ jobs and lives; but in a way, they’re doing us a service by opening our eyes to the vulnerability of the Internet.
Now what can we do to fix it?
What can we do? How about the “hackers” grow up and stop acting like 15 year old kids who dont have jobs or make their own money and are sitting on mom and dads comp doing this? Its DDos attacks, not real hacking guys. And seriously they are only asking for 5 BTC? really? This is just a show of “oh look what i can do, oh well i can do it more” GTFO
Do you really think this is a practical suggestion?
“How about the “hackers” grow up and stop acting like 15 year old kids who dont have jobs or make their own money and are sitting on mom and dads comp doing this?”
They’ll answer: “How about no.”
And then they’ll proceed to ramp up their attacks out of spite. It’s easy to demand everyone act civil. Hell, that would be an elegant solution to most security problems.
But it’s better to discuss ways to raise the cost of attack, so that acting like [insert ageist comparison here] is no longer a worthwhile way to cause misery for others.
Like it or not, that’s our responsibility as security researchers.
So many sites down. Paypal, Wired magazine, etc etc.
I am losing money as I run an ecommerce store using Shopify (down) and Paypal (down).
It’s a strange coincidence that Hover DNS was down for same reason a week ago.
http://hoverstatus.com
We host a number of domains on Dyn and have seen sporadic outages all day. There was a lull from 10:00 to 13:00 and then we started seeing DNS failures happening again.
GitHub status page (https://status.github.com) added a chart “98TH PERC. WEB RESPONSE TIME” that covers the attack.
And just coincidently “heavily armed police” are reported outside the Ecuadorian embassy? Maybe the DDoS of sites in addition to Twitter are a smokescreen
For some reason (!) I couldn’t resolve http://t.co
So here’s the YouTube URL for Doug Madory’s talk at NANOG on DDoS:
https://www.youtube.com/watch?v=LFJzu0AFDpU
Everybody loves the cloud even though it makes no money. I’ll take my chances with hosting my own stuff.
You host in house? What’s your site I want to check it out!
I find it remarkable that these people can cause so many millions of dollars of damage, but then ask for 5BTC (~$3k). With their skills, they could make more money in a week at a major tech company.
I find it remarkable that these people can cause so many millions of dollars of damage, but then ask for 5BTC (~$3k) for compensation. With their skills, they could make more money in a week at a major tech company.
11:20 pst Twitter is down and net sluggish with renewed attack.