A massive and sustained Internet attack that has caused outages and network congestion today for a large number of Web sites was launched with the help of hacked “Internet of Things” (IoT) devices, such as CCTV video cameras and digital video recorders, new data suggests.
Earlier today cyber criminals began training their attack cannons on Dyn, an Internet infrastructure company that provides critical technology services to some of the Internet’s top destinations. The attack began creating problems for Internet users reaching an array of sites, including Twitter, Amazon, Tumblr, Reddit, Spotify and Netflix.
A depiction of the outages caused by today’s attacks on Dyn, an Internet infrastructure company. Source: Downdetector.com.
At first, it was unclear who or what was behind the attack on Dyn. But over the past few hours, at least one computer security firm has come out saying the attack involved Mirai, the same malware strain that was used in the record 620 Gpbs attack on my site last month. At the end September 2016, the hacker responsible for creating the Mirai malware released the source code for it, effectively letting anyone build their own attack army using Mirai.
Mirai scours the Web for IoT devices protected by little more than factory-default usernames and passwords, and then enlists the devices in attacks that hurl junk traffic at an online target until it can no longer accommodate legitimate visitors or users.
According to researchers at security firm Flashpoint, today’s attack was launched at least in part by a Mirai-based botnet. Allison Nixon, director of research at Flashpoint, said the botnet used in today’s ongoing attack is built on the backs of hacked IoT devices — mainly compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies. The components that XiongMai makes are sold downstream to vendors who then use it in their own products.
“It’s remarkable that virtually an entire company’s product line has just been turned into a botnet that is now attacking the United States,” Nixon said, noting that Flashpoint hasn’t ruled out the possibility of multiple botnets being involved in the attack on Dyn.
“At least one Mirai [control server] issued an attack command to hit Dyn,” Nixon said. “Some people are theorizing that there were multiple botnets involved here. What we can say is that we’ve seen a Mirai botnet participating in the attack.”
As I noted earlier this month in Europe to Push New Security Rules Amid IoT Mess, many of these products from XiongMai and other makers of inexpensive, mass-produced IoT devices are essentially unfixable, and will remain a danger to others unless and until they are completely unplugged from the Internet.
That’s because while many of these devices allow users to change the default usernames and passwords on a Web-based administration panel that ships with the products, those machines can still be reached via more obscure, less user-friendly communications services called “Telnet” and “SSH.”
Telnet and SSH are command-line, text-based interfaces that are typically accessed via a command prompt (e.g., in Microsoft Windows, a user could click Start, and in the search box type “cmd.exe” to launch a command prompt, and then type “telnet” to reach a username and password prompt at the target host).
“The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present. Even worse, the web interface is not aware that these credentials even exist.”
Flashpoint’s researchers said they scanned the Internet on Oct. 6 for systems that showed signs of running the vulnerable hardware, and found more than 515,000 of them were vulnerable to the flaws they discovered.
“I truly think this IoT infrastructure is very dangerous on the whole and does deserve attention from anyone who can take action,” Flashpoint’s Nixon said.
It’s unclear what it will take to get a handle on the security problems introduced by millions of insecure IoT devices that are ripe for being abused in these sorts of assaults.
As I noted in The Democratization of Censorship, to address the threat from the mass-proliferation of hardware devices such as Internet routers, DVRs and IP cameras that ship with default-insecure settings, we probably need an industry security association, with published standards that all members adhere to and are audited against periodically.
The wholesalers and retailers of these devices might then be encouraged to shift their focus toward buying and promoting connected devices which have this industry security association seal of approval. Consumers also would need to be educated to look for that seal of approval. Something like Underwriters Laboratories (UL), but for the Internet, perhaps.
Until then, these insecure IoT devices are going to stick around like a bad rash — unless and until there is a major, global effort to recall and remove vulnerable systems from the Internet. In my humble opinion, this global cleanup effort should be funded mainly by the companies that are dumping these cheap, poorly-secured hardware devices onto the market in an apparent bid to own the market. Well, they should be made to own the cleanup efforts as well.
Devices infected with Mirai are instructed to scour the Internet for IoT devices protected by more than 60 default usernames and passwords. The entire list of those passwords — and my best approximation of which firms are responsible for producing those hardware devices — can be found at my story, Who Makes the IoT Things Under Attack.
Update 10:30 a.m., Oct. 22: Corrected attribution on outage graphic.
Brian – Very well written. I wrote about a similar challenge in June 2016 where massive number of DVRs were infected and were using SSH / Telnet. Totally agree that this is an archaic way to connect. https://techcrunch.com/2016/06/26/how-the-insecurity-of-things-creates-the-next-wave-of-security-opportunities
Do you think the regulators need to step in? Thanks for all the good work you are doing to raise awareness.
I think everyone here misses the point when you talk about finding ways to secure the ports. The way to solve this problem is very easy.
Just put a new govt agency like the NSA in charge of scanning for these devices like the hackers that perpetrated this attack. Since this is now a matter of national security this shouldn’t be an issue.
So the NSA will scan for devices. They will then notify the manufacturer if it’s a firmware security issue or the owner of the device if it’s a user name and password issue. If the manufacturer or owner does not secure their device in the amount of time, say 30 days, the NSA will brick their device.
See, simple. Both manufacturers and owners now have an incentive to make sure that their devices are secure. And this will remove all insecure devices from the internet.
Obviously, each country’s version of the NSA will have to do this within their own country. But the governments of all nations should coordinate on this effort to eradicate insecure devices from the internet.
Why not change the password and then notify the user 🙂
From what I am reading it may not be that easy to ‘fix’ these devices (or brick them). From what I am seeing it appears that these devices are embedded linux devices with the custom programming and OS code hard coded into onboard chips. In higher end devices they would use EEPROM chips and include the circuitry that would allow them to be reprogrammed. Unfortunately, from what I am reading I have seen several reports that indicate that reprogramming either the OS or the custom app(s) is not possible on these devices. I have not seen a technical discussion that goes into the whys and why nots but I am assuming that it was cheaper to assemble the boards without including the chips or logic that would allow the devices to be re-programmed on the fly. This is probably why the bad guys could not make their malware modification permanent. This is why powering off the device clears the malware.
If my assumptions are correct then the only way to make them more secure would be to get them into a repair depot and reprogram or replace the chips. Of course, if the devices were reprogrammed like this there is no guarantee that they would be able to patch all the holes or find all the back doors. This would leave the devices vulnerable to the same kind of attack once they got back int he field.
I agree.
Wikipedia: October 2016 Dyn cyberattack
https://en.wikipedia.org/wiki/October_2016_Dyn_cyberattack
It would be interesting to pick up one of those cheap camera’s and intercept the traffic with wireshark.
You might be interested in this article from SANS, where they connected a vulnerable DVR to the internet.
The short answer is: it won’t last long, a few hours at most, and it’ll get probed within a couple of minutes.
It seems like we need to add a category to UL listing or just anything that is imported in to the US to have something like a certified netsafe listing or certification. I would not suggest a government entity as they are just a nightmare but make it so consumers can know if the device they are purchasing has at least basic security function.
The other thing that needs to happen is for the consumer router manufacturers to have a standard centralized and dynamic resource that can help them recognize the latest threats to help consumers protect the net with their own routers. They could easily allow a router to text to their phone with potential threats.
The problem is that all devices will eventually fall to attack. Security is a set of best practices to minimize attack, but if anybody trys to tell you that their specific product cannot be hacked, walk away.
The problem involves providing secure/signed firmware upgrades over time to evolve the hacked products to prevent the spread of these ‘robots’. Often, as part of a hack on an embedded device, part of the hack will be to disable firmware updates.
It is a complex issue, but the biggest problem right now is that many of the vendors are not even trying.
Brian, and others. I have a fair feel for the technical matters involved. What I do not grasp is how the money flows. What is the objective, if not money, and how does the objective become fulfilled?
Exactly! Absent from all the technical explanation is the question of WHY would one want to DOS Dyn in the first place? What can be gained from an attack like this?
Extortion is a common reason. A company will receive a message that they pay up or they will be shut down. They will refuse to pay. When that occurs the DDoS begins. The company then shuts down vulnerable services, or get on DDoS mitigation like Verisign. So the attacker moves to the next level, and hits the data center the company is located in. Resulting in an outage for anyone located in that data center.
Extrapolate that to a backbone provider, and now you have a pretty good idea what happened.
It could easily be seen as a political statement.
If the IoT devices are unfixible, why can’t a White Hat find the devices and strip them from the botnets using the same vulnerabilities?
Thats maybe possible. But if the user/pass are hard coded. He must destroy ssh/telnet services. If we are talking about Mirai , as today, a power shut down is enough to clean the device.
Mirai is as far as I know, using BusyBOX/OpenWRT, x86/ARM.
That would be illegal.
Someone could take the Mirai source and with little effort modify it to instead seek out vulnerable devices and stop their SSH and Telnet services.
This would only be a bandaid on the problem as every time the device is restarted the services would need to be stopped again.
Also, it’s probably just as illegal since you’re commandeering equipment that doesn’t belong to you in order to implement this fix, so I wouldn’t recommend anyone actually do this.
A legal alternative might be for ISPs to scour their own networks for known vulnerabilities like this and alert their customers to the problem. They could even block those services until remedial action is taken. This would benefit both the ISP and the client, so it could happen.
Why shows the stats from kaspersky cybermap a botnet activity from yesterday which is even lower then the day before?
https://s14.postimg.org/3mw01uai9/bot1.png
On http://www.digitalattackmap.com/ there is no data from yesterday, 21 oct., (maybe not archived yet….)
Are we talking about facts or speculation?
Data 21.10.2016 is on-line.
http://www.digitalattackmap.com/
No special/strange kaspersky cybermap botnet activity these day.
No proof at all Mirai /IoT causes this attack.
For 90% I am sure.
I dare to call this a False Flag, inside job or human failure.
Tell me I am wrong, I am always willing to learn.
But no small talk. I want to see data or screen shots and no animations or a pictures from downcollector.
It’s stylish these days to claim everything’s a “false flag” attack. This seems more like a “proof of concept” attack.
For everyone cheering on the “Internet of Things”, I’d suggest that at some point, some large player will again find it convenient to use one of these attacks for political purposes. In the past, they shut down websites – with the Internet of Things, they could actually start causing the types of disruption currently associated with a bombing campaign. Imagine a piece of malware that could, say, brick all the self-driving cars in a given area on a specific date and time… or cause them to all lock up their brakes and steer hard left upon reaching 100 KPH or higher. Or simply turn off all the refrigerators in homes and businesses.
Perhaps its just a coincidence that this attack happened right after China objected to a US Navy vessel traversing the South China Sea. As DDoS attacks via botnets are typically anonymous, and don’t require a nation-state’s resources, who can say whether someone is trying to stir up trouble between the US and China, or whether the Chinese were registering a mild complaint, or whether someone was just testing an attack vector and messed up on the scope of their test, or…
It is stylish theses days to talk around and to make quick “conclusions”. The next, risky, step is for example to blame directly Russia.
Doesn’t matter how to call it. I want to see (data) from the “attack” (norse was down).
Why is there nothing unusual on kaspersky cybermap botnet activity graph (21.10.2016)?
Why is there nothing visible on http://www.digitalattackmap.com/ (21.10.2016)?
I really like the idea of an Underwriters Laboratories (UL) kind of certification authority.
UL exists, though, at the behest of underwriters: insurance companies. They can compel their customers to buy UL-approved products by saying “we won’t pay your fire insurance claims if you buy unapproved junk and your house, or store, burns down.
The path to establishing a UL for internet equipment involves a bunch of lawsuits against the CUSTOMERS of this bad vulnerable equipment, and massive payouts by their insurers.
So far, we have two identified victims of this latest botnet. One is Brian (and Akamai) and the other is Dyn. If these victims were to sue the distributors and makers of these IoWJ (Internet of worthless junk) products, it would be a step towards UL regulation. But such lawsuits are a long shot.
An industry “IoT” apprival mark like “UL” is laughable. Every day devices are sold with counterfeit UL marks. Witness tbose little power plug-packs that power phones, tablets, cameras, printers. Search YouTube to see takeapart videos of these lite fire hazards, all prominently marked “UL” and “CE”.
Hmmmm. If Miria source is available now, why not write a Anti-Miria version that takes over and blocks Miria or even disables IoT devices taking them out of the pool of potential bots (at least til the next reboot).
Brian, how do we know if our house full of IoT things is not part of the problem?
We have multiple brands of connected devices and a camera system and have changed passwords etc, but the vulnerability seems to be the hard coded credentials in the firmware. Is there a list of vulnerable devices? Maybe a tool to scan our network for affected devices?
Thank you for your stories and hard work.
https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/
Having worked as an engineer in a system integrator business for CCTV and physical security, I’m not sure that we should so hastly cast blame to China and skip the users and system integrators.
These devices are well known to be vulnerable and insecure by default in the industry. They are designed to be behind a firewall with ports forwarded to RTP/RTSP and/or web server whose password is changeable by the user.
Sure, having open Telnet and hardcoded passwords is unforgivable. Especially since SSH/Telnet isn’t even used by users or system integrators but authorised servicemen. But there are no good reasons for either users themselves (if selfinstalled) or system integrators to have these devices on DMZs or open internet, and every manual of every such device I’ve ever seen is pretty clear on running the device behind a firewall with port forwarding.
In 1983, I was Technical Editor of Digital Audio magazine, and when the first CD players began coming in for review, I found that they spewed out enormous amounts of RF interference that disturbed both radio reception and made TV unwatchable. I got in touch with the FCC and they were not aware of the problem because it was so new, but they promptly cracked down on CD players that did not meet US standards for emissions. Imagine if we had allowed such players to enter the country by the millions! Now, the FCC does not have the resources to test all the devices entering the country, because of aggressive downsizing and defunding efforts by the right-wingers.
How about a change in all the antivirus programs (incl Windows DEFENDER) that scanned all addresses on your local subnet for issues and reported them to the user?
And even though it might be legally questionable, how about using a reverse-engineered version of Mirai to change the network settings of any offending device – essentially disconnecting them from the internet – at least until a reboot. The owner would then have two options:
1. Contact the manufacturer – who could either send a “fixed” replacement, or refuse to make a replacement and be ostracized.
2. Throw the equipment away and buy a replacement.
In either case, any manufacturer that sold equipment that had weak or no security would be severely punished. And while it is true that consumers would pay as well – most of the ip cameras mentioned are not terribly expensive. In some cases, it may be possible to field-update firmware.
I see the discovery of this flaw as good news.
I don’t know how, and not able to back up this idea, but my gut reaction is that anything like this leaves a pattern, and can be blocked by some engineering, and tracing to the origin make the next people trying to use this exploit vulnerable.
Many of the owners of these devices get their TV service and internet service as a package. If the ISP flashed a 30 second warning on their TV screen every time there was a malicious packet from their IoT device, it would get attention.
ISP/routers should be able to filter at the source for MAC addresses in the range assigned to the manufacturer. That or as previously suggested someone needs to pwn those damned cameras!
IoT should be link-local only. Turn off the support for gateway device. Setup a local gateway service that can run a real, secure, server that is patched to deal with coordination.
Doing it all locally like that is not only ‘old school’ but is the exact opposite of what the intended purpose is. Doing this defeats the cloud.
I would agree that this would be better but too many people want cloud. Too many people want an app that goes through some other company as a man-in-the-middle between your home and your phone/watch/tablet/laptop. Too many people think that there is an app for everything and in that app we trust.
There is no reason why we can’t run a server in the home that will deliver things outbound only via password. But most people see such things as stuff that only the super rich or super smart can ever have. It isn’t true of course but that is how people see it. No one wants to understand these machines. But they will pay through the nose with money they don’t have to get someone else to fix them.
lol, people actually think that Apple invented the touch screen when they gave us the iphone.
Right about the touchscreen, but, like usual, you expect every owner of a device to know about all the issues of the device. To be an engineer qualified to improve the device. But , less then one third of the devices are bought by the young, engenders. Most are bought and used by those over 35. Who do not have security professional in their education. I would hazard a guess, those under 35. Have even less security training. Remember how much time the security professional spent on balancing a checkbook, or how to stop blood flow from a wound? And now you say, the owner has to clear the problem? Interesting.
The title of “Security Professional” is hardly required to take care of certain things. This is precisely the kind of thinking that frightens users into avoiding any real learning.
One does NOT need a bachelors degree to make adjustments to the settings within routers. One does not need to bachelors degree to install NoScript or add entries to a Host file. One does not require years of intense training to understand that the use of “toolbars” is a bad thing or that Flash and Java should be removed from their machine.
This is like suggesting that a Security Professional is needed before someone knows how to install apps to their phone. Wether these apps are good or bad is irrelevant. People DO know how to do these things already. Just like people already know how to link up to networks using WiFi.
This is ridiculous. We are not talking about rocket science. This kinda thing is WHY people don’t know things. There are too many other people willing to coddle them.
Each DDOS seems to be stopped within a few hours or at most a few days. What is the magic applied in each case that can not be developed into a general solution? (and no, I don’t think that the solution was to patch all XiongMai devices) Sorry no security expert here.
Glad to see the UL statement, we need to vet these devices as we do for the everyday electronics.
This sounds a lot like the problem of spam email, essentially innumerable amounts of unsolicited, fake, spoofed email sent to a target. The answer there was SPF and DKIM, albeit neither are 100% effective.
The difference with IOT is that single users do not necessarily have a fixed point (IP, DNS) for the end systems to validate against, thus no way to easily filter legitimate traffic.
Perhaps the shift to IPv6 is the answer. IPv4 traffic can be relegated to a deprecated public network with diminishingly low signal to noise ratio due to increasing frequency of unstoppable DDOS attacks, and IPv6 addresses can be permanently assigned and the owners held accountable for the traffic. Isn’t that how it works with passports, credit ratings, drivers licenses, and criminal records?
We’re not going to fix the IoT – just not plausible. These are plug and play devices for a reason. There’s no way my father is going to update his home security camera – just not going to happen.
The solution is a better backbone – as we saw, while Dyn went down, other DNS providers were up. It is up to the commercial decision of the clients (twitter, Spotify, etc) to determine if it is in their interest to have fail over / alternate DNS providers. Those that did had a short outage yesterday. Those that didn’t had a longer outage.
Far more practical and effective for those with a commercial interest in preventing their site from being knocked offline to solve the problem, than expect millions of tech illiterate people to do security updates on their DVR…
That’s not a solution – at best that’s a band-aid, and an expensive one at that. And with these devices being added at an astonishing rate, the attacks will become ever larger over time.
There are too many actors in all of this that just want to point fingers and not step up and try to solve the problem. As long as this continues, the problem is only going to get worse.
The team at ProjectVeritasAction.Com are researching these types of illegal activity.
IoT is a trillion dollar business, once again business will take priority over security, privacy, and national security interest. Developing a scan profile able to detect vulnerable host for this and future hosts with similar vulnerabilities will be worth the effort. Shutting systems down and rendering them useless will be the challenge. Who (country/agency) will own that power, who is willing to share that power , more important, WHO can be trusted with that power? This not only a technical issue, it is a problem on many levels. Solving it will not be easy, this is the tip of the iceberg.
Brian,
Regarding an industry security association, do you think this is something IEEE could handle if they wanted to? To me they seem to be an obvious choice for oversight if it’s within their capabilities.
Keep up the great work.
“It’s remarkable that virtually an entire company’s product line has just been turned into a botnet that is now attacking the United States,”
Yes, very much so. So much so that it is unlikely that this is something not planned. Makes me think that somebody has been reading Sun Tsu’s “Art of War”. And somebody wrote recently that there has been probing going on as if to test the whole internet system to see what would be the best way to disable it effectively.
One solution to prevent illicit use of IoT devices is for ISPs to block the protocols (Telnet &/+ SSH). I switched ISP because of a local provider doing so with SSH. That would hurt some individuals/companies that use SSH, but that is something we have to accept in the interests of national security, and can be worked around with a VPN.
Something needs to be done with the quickness, because what was rare is now becoming common, and there are strategic interests involved, and not just for America…
>One solution to prevent illicit use of IoT devices is for ISPs to block the protocols (Telnet &/+ SSH). I switched ISP because of a local provider doing so with SSH.
Couldn’t you just use a different port? Or were they doing some kind of packet inspection?
They were doing packet inspection and looking at the headers. Fairly simple to do technically.
So you say “punish all users” – not the manufacturers, poorly-trained installers and ignorant users of these devices?
Nice one…
I totally understand. I’m one of those users myself, and SSH is how I maintain/update my hosted websites and access my local server. However, unlike the IoT devices, I use greater than 2048 bit keys. If those IoT manufacturers where in Western countries, then maybe we could crowbar them into compliance. However, with them being in countries that operate at cross-purposes to Western interests, it’s not a likely scenario that we will be able to cause the manufacturers of the components to do anything.
This is not a convenience issue, it’s a strategic issue, IMHO.
This seems to be a viral trojan pushing a DDOS payload/service. If so, so far, within my envelope of expectable.
Obvious the interest to stop or meaningfully slow the spread, and to remediate the physical environment that resulted from an unfortunate confluence of reasonably expectable effects in, and of, the market in which everything is held possible.
Interested in learning just how powerful such a tool can be made. Time will tell.
I wonder whether the doubly-compromised devices need all the bandwidth available to them in practice, and whether much is needed at all for Mirai’s current purposes. Is there a means, external to the device itself, to cap each device to the absolute minimum for the intended purpose of each… ?
If such a means is practically available, would its resort, over time, encourage a more narrowly-defined set of devices to serve as the target of evolutionary iterations of Mirai: namely, those IoT devices with highest bandwidth ceilings?
Splitting the field by up-front right-sizing everything, but without actually imposing a burden on any device’s use, creates a tool– a lever, handle, or gate. Maybe it’s livestock management.
In any case, a more fruitful interpretation of the anticipated rejoinder could begin immediately there’s a signal.
Who can answer this question:
What was this attack intended to distract us from?
Power.
It’s some of the worst written C code I’ve ever seen but apparently it works.
It’s based on old code and not written from scratch.
Do you think he wanted to win an ioccc contest?
Wikileaks tweeted a thank you to the supports of Assange who was about to be arrested by an armed assault on the Ecuadorian Embassy. http://www.express.co.uk/news/world/723992/Julian-Assange-wikileaks-hack-twitter-amazon-spotify-obama
One wonders what the next level would consist of.
Botnets have been in use for quite some time and back in 2013/14 were used to massively mine bitcoins.https://www.wired.com/2014/07/how-hackers-hid-a-money-mining-botnet-in-amazons-cloud/