A massive and sustained Internet attack that has caused outages and network congestion today for a large number of Web sites was launched with the help of hacked “Internet of Things” (IoT) devices, such as CCTV video cameras and digital video recorders, new data suggests.
Earlier today cyber criminals began training their attack cannons on Dyn, an Internet infrastructure company that provides critical technology services to some of the Internet’s top destinations. The attack began creating problems for Internet users reaching an array of sites, including Twitter, Amazon, Tumblr, Reddit, Spotify and Netflix.
A depiction of the outages caused by today’s attacks on Dyn, an Internet infrastructure company. Source: Downdetector.com.
At first, it was unclear who or what was behind the attack on Dyn. But over the past few hours, at least one computer security firm has come out saying the attack involved Mirai, the same malware strain that was used in the record 620 Gpbs attack on my site last month. At the end September 2016, the hacker responsible for creating the Mirai malware released the source code for it, effectively letting anyone build their own attack army using Mirai.
Mirai scours the Web for IoT devices protected by little more than factory-default usernames and passwords, and then enlists the devices in attacks that hurl junk traffic at an online target until it can no longer accommodate legitimate visitors or users.
According to researchers at security firm Flashpoint, today’s attack was launched at least in part by a Mirai-based botnet. Allison Nixon, director of research at Flashpoint, said the botnet used in today’s ongoing attack is built on the backs of hacked IoT devices — mainly compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies. The components that XiongMai makes are sold downstream to vendors who then use it in their own products.
“It’s remarkable that virtually an entire company’s product line has just been turned into a botnet that is now attacking the United States,” Nixon said, noting that Flashpoint hasn’t ruled out the possibility of multiple botnets being involved in the attack on Dyn.
“At least one Mirai [control server] issued an attack command to hit Dyn,” Nixon said. “Some people are theorizing that there were multiple botnets involved here. What we can say is that we’ve seen a Mirai botnet participating in the attack.”
As I noted earlier this month in Europe to Push New Security Rules Amid IoT Mess, many of these products from XiongMai and other makers of inexpensive, mass-produced IoT devices are essentially unfixable, and will remain a danger to others unless and until they are completely unplugged from the Internet.
That’s because while many of these devices allow users to change the default usernames and passwords on a Web-based administration panel that ships with the products, those machines can still be reached via more obscure, less user-friendly communications services called “Telnet” and “SSH.”
Telnet and SSH are command-line, text-based interfaces that are typically accessed via a command prompt (e.g., in Microsoft Windows, a user could click Start, and in the search box type “cmd.exe” to launch a command prompt, and then type “telnet” to reach a username and password prompt at the target host).
“The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present. Even worse, the web interface is not aware that these credentials even exist.”
Flashpoint’s researchers said they scanned the Internet on Oct. 6 for systems that showed signs of running the vulnerable hardware, and found more than 515,000 of them were vulnerable to the flaws they discovered.
“I truly think this IoT infrastructure is very dangerous on the whole and does deserve attention from anyone who can take action,” Flashpoint’s Nixon said.
It’s unclear what it will take to get a handle on the security problems introduced by millions of insecure IoT devices that are ripe for being abused in these sorts of assaults.
As I noted in The Democratization of Censorship, to address the threat from the mass-proliferation of hardware devices such as Internet routers, DVRs and IP cameras that ship with default-insecure settings, we probably need an industry security association, with published standards that all members adhere to and are audited against periodically.
The wholesalers and retailers of these devices might then be encouraged to shift their focus toward buying and promoting connected devices which have this industry security association seal of approval. Consumers also would need to be educated to look for that seal of approval. Something like Underwriters Laboratories (UL), but for the Internet, perhaps.
Until then, these insecure IoT devices are going to stick around like a bad rash — unless and until there is a major, global effort to recall and remove vulnerable systems from the Internet. In my humble opinion, this global cleanup effort should be funded mainly by the companies that are dumping these cheap, poorly-secured hardware devices onto the market in an apparent bid to own the market. Well, they should be made to own the cleanup efforts as well.
Devices infected with Mirai are instructed to scour the Internet for IoT devices protected by more than 60 default usernames and passwords. The entire list of those passwords — and my best approximation of which firms are responsible for producing those hardware devices — can be found at my story, Who Makes the IoT Things Under Attack.
Update 10:30 a.m., Oct. 22: Corrected attribution on outage graphic.
I’ve been wondering if DDoS’s like this could make a mess of bank infrastructure, or computerized Wall Street trading. One thing is for sure, this is the “new normal”
Is there a convenient way for technically-capable consumers to scan devices on their home networks for susceptibility to Mirai infection? If I can clean my own house I’d like to go ahead and get started.
yes, just change the default password on any device that needs internet access. Problem solved.
learn to read. they can’t be changed. its not a user accessible password, no one can change it. thats the problem
Please re-read. The passwords are hard coded for SSH and Telnet access, and are not changeable by the user. Unless you are capable of rewriting the firmware, the problem remains after changing the Webserver password. Device will always be vulnerable. If the device is already compromised, unless you can reload firmware, it will remain ever part of the botnet.
Ken,
The easiest scanning tool to use would be nmap
https://nmap.org/
It can scan an entire network but it may not be able to give you an accurate picture of all the vulnerabilities.
Instead you might want to check out Nessus Home Edition.
https://www.tenable.com/products/nessus-home
Depending on how many devices you want to scan you may want to get a free trial of the Profesional version to do a one time scan.
Also don’t forget to scan the external face of your ISP Router which will require you to
a) Know your IP address
b) Go somewhere else to perform the scan
Neither of Josh’s comments are complete. The vulnerability is simple — SSH (port 22) or Telnet (port 23) is left on with a set of default passwords. You could use nmap to try and scan your network for either open port, but you would still need to try and check the passwords on your devices. The easier approach would be to look at the list of devices mentioned in an earlier article and if you have any of them take further action.
“Go somewhere else to perform the scan” really means just connect to the internet through a different network. If you can tether to your phone, that’ll suffice. Location isn’t important – it’s getting out from behind your ISPs connection
Probably best to look at the list of vulnerable devices and see if you have any of those. Make sure EVERYTHING on your network doesn’t have default password.
Ken,
I don’t have a vulnerable device to try. However, not a scan; but, a manual method is to try to login by telnet (explained in the article) using the user/password combo of the type of device being checked. Combos provided here:
https://krebsonsecurity.com/wp-content/uploads/2016/10/iotbadpass-pdf.png
For a dvr there are presently four combos to try.
Note: h.264 – Chinese dvr = dvr with XiongMai Technology board
Isn’t it a case of configuring your router so that, for instance, Telnet and SSH requests cannot get through to your home network? Likewise outbound “phone home” type messages could be blocked by your router (if we knew where to find black-lists to load into our router’s firewall settings!)
If you have a secure network, it is far less worrying what is running within that network?
Points again to most of us needing to take a far more active interest in our router. We do system updates on our PCs, most of us have security packages on our PCs which auto-update several times a day and many of us run specific scans on our PCs to check for particular nasties. Yet we buy a router, plug it and and just about forget about it!
It’s like not maintaining your front door!
The map should be attributed to downdetector.com and openstreetmap contributors, not Level3.
And it’s apparently a map about people reporting Level3 outages, which is probably a misleading measure of the current DDoS attack – number of people who reported that their Level3 service doesn’t work when in fact their Twitter/Reddit/etc doesn’t work..
See http://downdetector.com/about-us/
What a mess. Thanks for keeping on top of this. We have a long road ahead of us with IoT.
Given that most home users will have their devices behind a router using NAT to reach the Internet, how do hackers reach those devices from the Internet?
1rst they are not all behind nat
2 anything within the same network can take control of it, like any unprotected pc or iot devices(such as but not limited to xp machines)
Once compromised , the connections is made from inside to the botnet, which is generally allowed by common nat routers.
Keep your great work up! This is getting more and more a huge…thing
Why not use the same code-base from Mirai to brick devices? That would create a “recall” of sorts.
Serious question….
In the case of Mirai the program lives in volatile memory and actively blocks attempts by other malware to gain access. So why not in fact set up a competing “benevolent” botnet that does nothing other than what Mirai does sans the malicious bit?
I’m very glad to see that krebsonsecurity(.)com stayed online today.
It was lagging earlier. Wasn’t fully down though, which is good.
Unfortunately today was just a proof of concept, operational test. More to follow.
Troll warning, kids be everywhere these days! Proof or GTFO.
@JS
That is thanks to our good ol’ friend uPnP. Which is typically enabled by default on consumer router/gateways. This allows devices to automatically open and forward ports necessary for web access.
so basically all products built around this company’s components should be tossed in the trash.
Yupe, they should be trashed. There is no other way to fix it. From what I read, end user will be very hard press to upgrade the firmware!
One thing we need to push is to have the router manufacturers to enable mitigation technologies on their firmware. If we can stop this traffic at the edge (of the source network) then we don’t have to wait until all the insecure IoT devices die of natural causes…which is the only way they will disappear. There is NO reason SSH and especially Telnet to be open inbound on anyone’s network unless they do so on purpose for (likely a poorly thought out) reason. Extending that even further, there is no reason non-standard ports should be open. Its not that hard to inspect outbound traffic at the router and block DDoS sources either. I think its pretty easily reduced if not resolved.
This makes sense. I picture a city with a congested downtown, say Madison WI. Do you improve the blindly build a 4 lane freeway directly to the downtown without thinking of, or taking any action to improve the city transportation system? Instant, predictable gridlock. The city blunders along, talking about rail for decades but doing nothing. I recall the huge benefit Bart provided in SF after the eartgwuutook out the roads, and the problems in NY when subway was flooded and damaged.
Another example. The state gov, or a big 6 consultant decided to centralize all the intake processing for health care applicants. So instead of little independent offices being able to directly talk to clients and resolve issues, in their quest for soviet style central control, and efficiency they funneled all the work into one overwhelmed office. I pointed out the flaws in their idea. Blank stares. Too late. So instead of efficiency, small errors would choke up the whole works, humming up the system with bad data, which was previously naturally throttled at the peripheral offices. Well, fortunately they kept the original system in place
It was kind of predictable and soon we will see this kind of attcks growing …
I had it mentioned in my recent article
https://twitter.com/akumar_net/status/789557097243418624
Any data on size of attack? The digital attack map is finally showing one 45-minute attack, but only 102G peak.
btw, Krebs site has appeared to be down for short periods today, which doesn’t bode well for the rest of us.. without Google’s massive bandwidth to absorb the attacks.
Are these IoT devices reachable if telnet port 23 is not on? I was trying to scan mine today.. but no ports were open.
did you disable upnp on your router? you need to be outside of your network to scan it.
It is time to make illegal selling a internet-attached device that is not supported with security updates. Write your Congress critters.
A few weeks ago, an ISP technician strongly advised me not to change the default login credentials to my new wireless router. It was so that they could “log in and help me troubleshoot issues” if needed.
Very shocked to hear an ISP would ever recommend such a practice given today’s insecure state of technology. Changing default credentials is practically “Security 99” (not even a level 100 class).
That depends on the router. Some routers do not have a ‘default’, commonly known login and password. Verizon FIOS has recently been sending out routers to replace older ones. The new routers have a ‘default’ password which is unique to that individual router.
What I don’t like about the new routers is that their admin password is shorter and simpler than their wi-fi password.
http://www.xiongmaitech.com/en/index.php/about/contact/42
Anybody feel like sending a couple of million phone e-mails?
Maybe Xiongmai should be taken off line til they fix this…
Brian, have you tried contacting them?
And boom. You just became that which you despise. And committed the crime of conspiracy. How does it feel to go from Holier than thou to criminal in a single posting??
http://www.xiongmaitech.com/en/index.php/about/contact/42
Anybody feel like sending a couple of million e-mails?
Maybe Xiongmai should be taken off line til they fix this…
Brian, have you tried contacting them?
Well, there is a botnet out there that could easily send millions of emails.
Hi Bryan,
As a long time reader & fan who has been a tech savvy holdout on IoT products, I am curious about your input regarding Apple’s HomeKit architecture.
Despite being highly attracted to home automation, I held off from deployment of such devices for many of the reasons that have suddenly become so apparent. I am comfortable with HomeKit.
By the way, I purchased three autographed copies of your book when first published. Are you making autographed copies available again for the new press run?
I can’t thank you enough for your incredible service to the global connected community.
Live long and prosper,
Joe Outlaw
If we’re able to identify the manufacturers of the infected devices, why are they not liable?
Brian,
What you have suggested is exactly what and why we have created the IoT Trust Framework. Today we are working with some major retailers as well as the Nat Association of REALTORS. This will help curb products coming into the market and strengthen the supply chain, but we also need to address the risk of these at risk devices (and their applications) that are installed. We may find the need to “brick” some of these devices for the good of the internet or urge companies to recall orphaned devices. More at https://otalliance.org/IoT.
Aloha Brian,
Contact UL to learn more about the new (still in the comment period) 2900 series certs. If mfr’s get on board, the program will help close this existing gap for commercial security products.
Availability and irresponsible use of poor equipment will continue to be an issue in the long-term.
Are there any indications that this event is related to recent comments by (I believe it to be) Chris Soghoian?
He had expressed a concern about a series of probes of a multitude of sites.
I found it very curious that nearly every link to the leaked transcripts of Hillary Clinton’s Goldman Sachs speeches stop working this morning. They worked last night but today they were broken…nearly every single one that I could find. Unintended consequence of today’s ddos attack, mere coincidence or intentional?
What OS do these insecure devices use? Can’t something be done at the OS level to prevent unauthorized use of the devices?
Linux.
And no.
This is a userspace misconfiguration. Telnet or SSH left enabled by the manufacturer either not requiring authentication or with known / documented credentials.
The devices may be sold special purpose devices, but that’s just a collection of applications built on top of a general, purpose computer. The OS has no way of knowing that a given application’s request is unreasonable.
In some rare cases, a human may purchase a component and repurpose it. If the device has artificial limitations, some customers would be upset…
For having clearly identifiable and highly centralized attack entry points, Dyn is as much to blame for this mess as Hangzhou Xiongmai Technology Co., Ltd. and their crappy vulnerable components.
If you don’t know where to shoot, you can’t hit your target (obscurity). If your target is a swarm, having a gun is useless (distribution).
Drone, respectfully, what you are basically saying is comparable to saying a victim of a mugging is as much to blame as the mugger. Yes, there are things that can be done to better protect against this, like carrying mace or not walking alone in a bad neighborhood. Still, the victim is not at fault. Please do not misunderstand me, there are probably things Dyn (and others) could and should do to better protect, but that does not mean it will stop. As long as you are connected to the internet, DDOS is a real threat. Network providers and ISPs should help as well. Also, obscurity is not, and never will be, security. We should make every effort to protect ourselves. At the same time, we still need to make it harder for this to be possible by not allowing these ridiculously insecure devices to be produced by vendors and to go after the criminals responsible. That is what the article is pointing out.
Scan for these devices as has already been done, log their IP’s, and have their ISP block all the upstream ports to those clients,
I’m amazed. About the “security research” as well as about the, pardon me, thoughtless “solutions”.
First, solving technical and social problems by committees has amply shown to fail. Besides, there *are* plenty standards bodies, mfr. assoc. etc, etc. – and their interest more often than not is *not* the users, clients, or communities well being but that of their industries, their sponsors and lobbyists.
Second: Rather than securing each and every small device, how about fixing them router/modem plastic boxen first? Oh yes, pardon me stupid; there seems to be no or too few mfr. assocs of plastic boxen and that’s the primary problem. But sure, repeat the futile attempts and hope …
Finally: Assuming for a second I were a chinese (or us-american or …) secret service colonel, I’d jump in joy and I’d certainly *not* want my mfrs. to tighten their products as that would mean to close a useful source as well as a useful attack vector against other countries where millions of those crap products are sold. “But your own citizens?!” I hear you asking … Counter-question: Do you know any secret service not spying on its own citizens?
And now, good luck with committees and associations …
The NANOG talk given by Dyn prompted some good comments in the Q&A afterward. The speaker who was talking about “common carrier” (Geoff Huston?) issues had a point for transit providers who are simply in the path but as another speaker pointed out, really do not apply for edge providers. Most of these attacks rely on an ability to spoof source addresses of packets Edge providers can and should filter these on ingress to their network. Even multihomed customers can be accommodated through proper use of routing registries and route /packet filter configs built using those registries.
There really is no excuse these days for a network accepting packets claiming to originate from an address range outside of the customer’s assigned address space other than laziness. We need to tighten up on edge provisioning.