A massive and sustained Internet attack that has caused outages and network congestion today for a large number of Web sites was launched with the help of hacked “Internet of Things” (IoT) devices, such as CCTV video cameras and digital video recorders, new data suggests.
Earlier today cyber criminals began training their attack cannons on Dyn, an Internet infrastructure company that provides critical technology services to some of the Internet’s top destinations. The attack began creating problems for Internet users reaching an array of sites, including Twitter, Amazon, Tumblr, Reddit, Spotify and Netflix.
A depiction of the outages caused by today’s attacks on Dyn, an Internet infrastructure company. Source: Downdetector.com.
At first, it was unclear who or what was behind the attack on Dyn. But over the past few hours, at least one computer security firm has come out saying the attack involved Mirai, the same malware strain that was used in the record 620 Gpbs attack on my site last month. At the end September 2016, the hacker responsible for creating the Mirai malware released the source code for it, effectively letting anyone build their own attack army using Mirai.
Mirai scours the Web for IoT devices protected by little more than factory-default usernames and passwords, and then enlists the devices in attacks that hurl junk traffic at an online target until it can no longer accommodate legitimate visitors or users.
According to researchers at security firm Flashpoint, today’s attack was launched at least in part by a Mirai-based botnet. Allison Nixon, director of research at Flashpoint, said the botnet used in today’s ongoing attack is built on the backs of hacked IoT devices — mainly compromised digital video recorders (DVRs) and IP cameras made by a Chinese hi-tech company called XiongMai Technologies. The components that XiongMai makes are sold downstream to vendors who then use it in their own products.
“It’s remarkable that virtually an entire company’s product line has just been turned into a botnet that is now attacking the United States,” Nixon said, noting that Flashpoint hasn’t ruled out the possibility of multiple botnets being involved in the attack on Dyn.
“At least one Mirai [control server] issued an attack command to hit Dyn,” Nixon said. “Some people are theorizing that there were multiple botnets involved here. What we can say is that we’ve seen a Mirai botnet participating in the attack.”
As I noted earlier this month in Europe to Push New Security Rules Amid IoT Mess, many of these products from XiongMai and other makers of inexpensive, mass-produced IoT devices are essentially unfixable, and will remain a danger to others unless and until they are completely unplugged from the Internet.
That’s because while many of these devices allow users to change the default usernames and passwords on a Web-based administration panel that ships with the products, those machines can still be reached via more obscure, less user-friendly communications services called “Telnet” and “SSH.”
Telnet and SSH are command-line, text-based interfaces that are typically accessed via a command prompt (e.g., in Microsoft Windows, a user could click Start, and in the search box type “cmd.exe” to launch a command prompt, and then type “telnet” to reach a username and password prompt at the target host).
“The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present. Even worse, the web interface is not aware that these credentials even exist.”
Flashpoint’s researchers said they scanned the Internet on Oct. 6 for systems that showed signs of running the vulnerable hardware, and found more than 515,000 of them were vulnerable to the flaws they discovered.
“I truly think this IoT infrastructure is very dangerous on the whole and does deserve attention from anyone who can take action,” Flashpoint’s Nixon said.
It’s unclear what it will take to get a handle on the security problems introduced by millions of insecure IoT devices that are ripe for being abused in these sorts of assaults.
As I noted in The Democratization of Censorship, to address the threat from the mass-proliferation of hardware devices such as Internet routers, DVRs and IP cameras that ship with default-insecure settings, we probably need an industry security association, with published standards that all members adhere to and are audited against periodically.
The wholesalers and retailers of these devices might then be encouraged to shift their focus toward buying and promoting connected devices which have this industry security association seal of approval. Consumers also would need to be educated to look for that seal of approval. Something like Underwriters Laboratories (UL), but for the Internet, perhaps.
Until then, these insecure IoT devices are going to stick around like a bad rash — unless and until there is a major, global effort to recall and remove vulnerable systems from the Internet. In my humble opinion, this global cleanup effort should be funded mainly by the companies that are dumping these cheap, poorly-secured hardware devices onto the market in an apparent bid to own the market. Well, they should be made to own the cleanup efforts as well.
Devices infected with Mirai are instructed to scour the Internet for IoT devices protected by more than 60 default usernames and passwords. The entire list of those passwords — and my best approximation of which firms are responsible for producing those hardware devices — can be found at my story, Who Makes the IoT Things Under Attack.
Update 10:30 a.m., Oct. 22: Corrected attribution on outage graphic.
I don’t believe Telnet or SSH qualify as obscure, millions of people use them every day.
It’s pretty dumb to not randomize the password before shipping, but this problem seems relatively trivial to fix.
They’re relatively obscure for most people buying and using these shoddily-secured IOT devices. Alas, the people who DO know what telnet and SSH are probably are not part of the problem.
But why are these devices exposed to the Internet?
I’d expect that most of them are behind ordinary NAT routers…
Mostly through the perils of UpnP so people can do things remotely, the most pointless I’ve seen lately is so they can beep there car horn at home from a flight to drive there family crazy…. What a world we live in. The proper use here would be CCTV, environmental controllers from home to farmers with poly-tunnels or remote stock management.
IoT devices are made quickly and cheaply with a focus on consumer ease, as we see yearly consumers do not like passwords and other forms of security.
>Inb4 why don’t ISPs just stop these devices connecting
Having worked in front line tech support for consumers there is no way the average consumer is taking having there service blocked, they’ll just keep moving around providers until they find the one that doesn’t care.
Yes, but many if them use uPNP to make a port forwarding on the router… Why they would expose a telnet interface that way is beyond my understanding but that’s the reason why I disable uPNP on every router when possible.
“but that’s the reason why I disable uPNP on every router when possible.”
Is that possible with phone company modems that they give you for internet access?
It doesn’t matter because the configuration is in the software – not the hardware.
Usually it is possible; I’ve found it to be possible with every DSL modem I’ve gotten from Verizon over the past decade, though you might have to hunt through “advanced settings” menus and click through a warning saying “don’t change these settings unless you really know what you’re doing” or similar.
I have heard of people issued modems by their phone company where it was locked down, but that hasn’t been my experience.
Easily done. Buy a router, run a cable from your ISP device to the router internet input. Now secure your router.
I had to do this when I bought an Apple Airport. It only has one device port available, and I needed to plug in three or so. My previous router had WiFi issues, so I turned off the radio and plugged it in to the Airport to use it as a switch.
Works fine.
Most of these devices don’t allow users to change Unix passwords from the web interface. Ideally, they’d disable remote logins over telnet or SSH or disable them completely, since they’re not necessary for the devices’ use cases.
Even if we did force these simple controls, we have a massive install base out there and no incentive to upgrade or fix them.
“this problem seems relatively trivial to fix.”
OK, there’s a million “smart” light bulbs already deployed around the world, running different versions of the same firmware, but under lots of different consumer-facing brands. Most people who have one don’t know what SSH is, and certainly aren’t aware they’re running 20 SSH servers in their house. What’s your “trivial fix”?
Can’t residential ISPs block telnet and ssh by default?
You would think that residential ISPs would block commonly used “inbound” ports, or at least the range formerly called “well known services” (0 through 1024).
Sadly, most residential ISPs do not make this effort. I know that my ISP does not make this effort. I have been seeing port 22 and 23 probes increase since the Mirai exploit was released. I am now seeing those probes “on average” about 2 every minute of the day. Fortunately I do not have “exposed” IoT devices. I have a firewall that I fully control.
The question is why are these ports open on IoT devices? Clearly if ISPs block the 0-1024 range manufacturers will just move to a higher range. This will only solve the problem temporarily.
I’m thinking something the other way around. If an ISP sees that one of their subscribers are participating in such an attack, can’t they take some action at that time, like block the traffic and contact the subscriber.
Going even further, if the victims of these attacks know all the IPs where the traffic is coming from (which I know is not always the case), then couldn’t some big-name website obtain that list and when someone known to be participating in the attack visits that popular website they see a banner that politely asks them to be a good internet citizen?
Just blocking inbound like this is a band-aid, and won’t really solve the problem. For one thing, you would be reliant on all ISPs performing this filtering, when in fact most ISPs can’t be bothered.
I suppose one thing they might do is block outbound traffic with obviously forged IP addresses. That might help some, but they could then just forge IP addresses that are within the ISPs subnet.
I suppose ISPs might reconfigure the end-user firewalls to disable UPNP by default. Who knows what sorts of problems that might create, but we are reaching a point where the internet is struggling to survive as it is.
actually, ISPs can, should, (and need to be required to) block any fake sender adresses from DSL and Cable subcribers.
Those would’nt stop infectrios, but would reduce the attack potential of those devices.
In this case, though, a simple direct brute force attack, it wouldn’t matter. unfortunately.
ISP are not going to block access to ports. This would effectively create a precedent for them to be more active in management when they are only interested in providing access, not security.
They aleady block port 25. What’s one more.
No, they don’t. Only some, do, and your remark is the perfect example of why even those few are a problem.
Destroying the internet, one port at a time, is not going to protect it.
Blocking ports mindlessly is not the answer. It’s like having a third party decide which TV channels should and shouldn’t be sent to your cable TV box.
Most ISPs have there own equipment still set to default user/pw. My local ISP has all their power supplies for their fiber boxes still set to the default user/pw so anyone could get in and shutdown the fiber converters without much effort. I have disclosed this to them but they still dont do anything to secure the devices.
What i am saying is dont plan on ISPs to help much..
That’s the core of the problem. Everyone is trying to pass the buck – everyone wants someone else to find a solution.
It may well be that massive outages like this may finally spur people to actually do something, but it isn’t clear what the something might actually be. The key is you can’t depend on all ISPs cooperating – some might, but many might not. You obviously cannot depend on all manufacturers of various IoT devices to not do something stupid either. So any solution really needs to have multiple layers.
Does anyone know what an end user might experience if they were browsing the web at the same time their IoT device was commanded to participate in a Mirai attack?
I’m assuming they wouldn’t notice it except for some sluggishness, unless they had some network traffic program that alerted them to an outbound stream above a certain level, etc.
Also, if the Krebs hit was 620 gbps and there were approx 515,000 infected devices, that’s 1.2 mbps on average. Does that sound right? I apologize if Brian already covered this previously.
Finally, does anyone know of a good traffic analyzer program? I use NetWorx but I’m guessing Mirai traffic wouldn’t have been picked up by this program had I been infected. Windows Resource Monitor seems pretty thorough but lacks notification features.
You’ll need something at the edge of your network to analyze traffic from other devices; your windows machine isn’t capable of seeing traffic between other devices on your network and the internet (with one unlikely exception)
You basically need a router that can tell you how much traffic is going in and out of your network. Ideally something that tells you how much internet traffic is coming from each device. dd-wrt and pfSense are both router OSs and have some capability of monitoring traffic.
Also, unless I’m mistaken (like if UPnP is at play), you would have to have your IP camera or DVR publicly exposed on the internet for your device to be hacked; you would have to either have your camera plugged straight into your modem, or have explicitly made an exception in your router to allow traffic to the camera.
UPnP almost certainly has to be involved and I’m surprised no one in the press seems to be calling that out. UPnP was always a terrible idea.
Thank you!
A packet sniffer can give you a pretty good idea what is going on, but you need to spend some time learning how to use it properly and how to interpret what you are seeing.
I was once at a conference, sitting in the back row, and I could see some guy a few rows ahead of me running a packet sniffer (WireShark in this case). I couldn’t tell exactly what he was doing since I wasn’t close enough, but it seemed like a rather odd thing to be doing in a roomful of non-tech conference attendees. At that point, I turned off the WiFi on my own laptop.
Is there any sound detail on motive or what industry will be targeted?
As far as I understand, each hardware vendor gets allotted a certain range of MAC addresses for their hardware. Can we just blacklist the offending vendor’s MAC mask in gateways and neuter their devices across the internet? Obviously they would no longer work but it seems like a small price to pay…
MAC addresses aren’t used in routing between networks. Routers use IP addresses when making routing decisions and the source MAC address will change from hop to hop as the packet traverses networks. Unfortunately, this won’t work
I think the MAC address can be spoofed as same as the IP address. If these hackers are capable of compromising this much number of IoT devices, they can spoof the MAC address too.
I am not 100% clear on the networking details of how these IoT devices are connected to the internet but I have to imagine that they are probably not behind any kind of router or firewall. I am guessing that the cameras and DVRs or whatnot must be connected directly to the gateway/cable modem. The gateway then acts in passthrough mode and every packet that comes in from the internet to the gateway’s address is then passed directly to that device.
Needless to say, this is not how I would setup any network I was involved with but I can see where a consumer (or a tech that doesn’t give a sh&t) might take the path of least resistance and just ‘plug it in’ and move on to the next job.
As far as solutions go… all this chatter about changing to a different DNS server is short sighted,,, If they have the ability to take down one of the primary DNS services then they have basically crippled the internet and they will probably be watching the traffic to see where the traffic is being re-directed to so they can pick a new target and broaden their list.
The long term solution has to involve either taking these IoT boxes off-line or crippling their command and control channel. The former is, in terms of sheer logistics, almost impossible… there are too many of them and there is no central registry of owners or locations. That leaves crippling the command and control channel. If they are in-fact getting into these devices by using the telnet and ssh ports to get to root on these devices then perhaps the solution is to change all the boundary routers to block the default telnet and ssh ports so as to make these packets ‘non-routable’.
This last solution is radical but if this goes on like this for another week the internet will be ready for it.
@Pleamonn Take a look at Schneier on Security for some ideas on the intent behind this attack: http://www.schneier.com/blog/archives/2016/09/someone_is_lear.html
I wouldn’t discount organised crime myself. As some of the commenters on that article state, it’s pretty self-defeating for China to take down the internet. But an organised crime group would be interested in doing it for extortion.
One motive I’ve heard for China performing attacks against Western sites, is to get Westerners to unwittingly extend the Great Firewall, without the domestic PR issues of actually blocking some Western sites themselves.
Basically, how many people automatically drop all packets coming from China? Realize that that means that Chinese people can’t access your site any more, and now you’ve extended the Great Firewall for them.
I am aware of several companies that do business only in the U.S. that drop all packets from China and certain other countries. There is no reason for someone from these countries to be accessing their network.
Telnet is not installed by default in Windows. It has to be installed separately from //add/remove windows components// from control panel
The article doesn’t say they telnet into Windows. They telnet into some router API. Once they have access to the router, they have the ability to poke around on the LAN.
I’ve been wondering about this for awhile. I always thought my router was my firewall, but apparently devices on the LAN can bore holes to the WAN without my permission. I guess I have some reading to do.
I’d like to see evidence of who these “state actors” are, before crying about the sky falling.
HEY! Where’s John Mcafee? he’s likely somewhere in Belize, running a warehouse of hackers.
Probably out of anger for losing the nomination for US President.
But, he’s far too smart to pull the #BlamePutin, BS.
It is easy to stipulate & point fingers at others. Everytime there is a cyber takedown it is usually the same people/companies who make the biggest noise in analysing and making vague conclusions without much proof. When the M25 comes to a halt because of a blackwall tunnel mishap these experts will anounce that at one least one Tesco juggernaut was seen leaving M25 near heathrow so Tesco is involved in the traffic jam !!!
Could this be remedied by mandating IPv6 and also randomizing addresses? Making scanning IP ranges to find vulnerable devices prohibitively expensive?
It would take a long time to scan 2^128 addresses to find a million or so potentially vulnerable devices.
This sounds good in principle, but then hackers would just target the active IPV6 addresses. When IPV6 becomes standard, a person won’t be able to just pick a random IP “out of the wild” so to say, unless they’re a tier-1 internet provider.
All this does is bring us back to the original problem if a hacker simply scans the IPV6 ranges that tier-1’s had allocated, rather than all the IPV4 addresses.
That would depend how many devices you had scanning…
One device finds another vulnerable device and infects it. Now you have two scanners, which partition the IPv6 space between them: you’ve gone from 2^128 time to 2^64 time.
Those two devices each exploit another two. Now you’re at 2^32 time.
Some IPv6 address spaces are known not to be allocated; remove those and you can save more time. Add in knowledge about how MAC addresses are generated and converted to/from IPv6 addresses?
Before long, the problem domain gets a whole lot smaller.
I believe that math is wrong.
If you’re scanning 2^128 addresses and you add another scanner
then each is scanning 2^127 addresses, not 2^64.
If you have 4 scanners each is looking at 2^126 addresses.
2^3 scanners have 2^125 addresses each
2^4 scanners have 2^124 addresses each
and so on …
Security by obfuscation rarely ends well.
Who has DIY scrubber for affected DVR devices?
You are also assuming folks have the IT knowledge to go in and make these modifications. I tinker but most of it is why beyond me. More mechanically minded. Most just plug and play their electronics.
Well done, you have given sites like the BBC the clickbait they love, and they are pretending ALL IOT devices are affected.
Whatever happened to journalistic responsibility!???
@Mark, if you have an actual beef with something I wrote, please do tell. My story was actually very specific about which IoT devices are the problem here, even linking to original analysis of said systems.
e.g.., https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/
As far as I know, there aren’t any other journalists digging as deeply into this issue. If BBC overgeneralizes the issue, take it up with them. But what’s that got to do with me?
Can you explain why you implicated IoT when these devices clearly do t fall under anyone’s idea of what an IoT device? Is this a tech-specific click bait? Does anyone else here think a security camera falls under the same category as a Arduino or a Raspberry Pi? …cause it flat out doesn’t.
I think networked security cameras share the same IoT label as Arduino and RPi. They’re sending digital information over the internet. Maybe it’s harder to get them to send whatever’s being sent in DDOS attacks. Maybe it’s impossible, I don’t know. But they ARE running software to enable them to do the job they do, and if it’s possible to modify that software over the network (something that’s true for every networked security camera I have experience with, even if the “modification” is only an IP address to communicate with) then in theory it’s possible to exploit them maliciously. Isn’t it? If I’m wrong, simply explain why.
They’re very much in the spectrum of IoT. Why would CCTV cameras not count when Refrigerators, toasters, thermostats, sound systems, etc all do?
Although, admittedly, having a list of the user:pass’ in your article seems to be an unwise decision 🙂
Probably a good thing this is happening now. If the powers that be do not do something to rectify the situation it could become prohibitively expensive, to the point of life and limb I am afraid. I never use UPnP but I do have an older wireless router I believe is secure, but will be retired by the end of the year, and not replaced.
What we need is a governing body – like the Linux Foundation – to make sure IoT devices are up to snuff (no known vulnerabilities, better password management, etc.) then place a stamp of approval on the box. We, as IT experts, tell users only to buy hardware with that stamp.
I think this is the only way to stop the IoT threat.
So the government knows who did this, but they are refusing to tell?
“Obama administration officials have determined the outages were the result of a malicious attack, according to a federal law enforcement official speaking on the condition of anonymity to discuss internal assessments. Investigators have come to a preliminary conclusion as to who carried them out, but are not planning to make that public for now, the official said.”
http://www.latimes.com/business/la-fi-tn-dyn-attack-20161021-snap-story.html
Everyone blaming ISPs for not doing something also need to realize that the ISPs are also handcuffed thanks to end users and past complaints of traffic blocking. It’s easy to day they can block something, but when somebody needs to use those ports for a legitimate purpose then there will be FCC complaints and fines. It’s a Catch 22 scenario.
Where would a MAC whitelist fit into this? Or would it?
Is there any information in the packets coming out of these IoT devices that would identify them as an IoT type of device? If so, the ISPs or the big internet routers could just dump that traffic into the bit bucket, which would make those devices useless outside of the home. We could then require new IoT devices to have the necessary security built in, such as a random long password unique to each device the owner cannot change. Unfortunate to brick the old devices but when you live on the bleeding edge…..
Quite ironic that governments (as indicated by actions of the FCC and similiar organizations) are more worried about users themselves taking complete control of their hardware, and thus gaining the potential of violating frequency spectrum regulations, do apparently do nothing about users being locked out of their devices, preventing them from _avoiding_ criminal acts being commited by others by making use of the hardware they own.
could one use the Mirai toolkit to close all ports or reboot on a corrupted /boot or cause overheating to all those infected IoT devices? I mean: it’s a formidable way in to break it!
I used to install CCTV over IP devices, high end and cheap stuff. What i have seen is that much of the CCTV installers had little or no experience in network management or security. I have been to industry training where the installers were shown how to dump the entire CCTV system into the DMZ on the router as its easier to set up proper port forwarding and firewall rules.
I’d say this is a big part of the problem with these devices
I hate to say this as an option, but is it time for a white/grey hat effort to run the same exploit on these cheap IoT devices? Instead of doing it to create a malicious botnet, do it to randomize the password and harden it’s security options?
Granted, I’m sure there’s physical ways to reset it to the horrible manufacturer default, but I’m just brainstorming here. Since no one seems to want to take responsibility for these horrible cheap devices out in the world, someone needs to take the offensive to keep them out of the black hat’s hands.
The way I see it, if you are a business with high availability requirements you’d better have a good HA-DR solution ready to roll. I know there were several banks affected, unable to provide wire services, etc. One we work with was able to get back up and running using old school P2P tech. Slow and manually intensive, but working just the same.
This may enable Nation States to launch against others 1st & disrupt infrastructure, enabled by a Chinese Mfg? Just saying there is a bigger risk than usual DDOS targeting.
Hi Brian – excellent post as always.
What specific OS’s are compromised? Everything I see says what Mirai does, but not to what OS’s.
Thanks.
All of them run on some form of linux, including mobile architecture like Arm, AFAIKT.
It wasn’t hackers. Well….they were, but they weren’t ….unsponsored.
These shoddy devices will end up doing way more damage then the Samsung Galaxy Note 7 ever could have done. Why are we allowing these insecure devices to be sold? It’s completely unacceptable and like you said the companies should be forced to recall them and bear the cost.